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(Project  No.  6AE-0067.00) 

Live-Fire  Test  and  Evaluation  of  Major  Defense  Systems 

Executive  Summary 


Introduction.  The  DoD  defines  live-fire  test  and  evaluation  (LFT&E)  as  full-up  (folly 
loaded  or  equipped),  system-level  survivability  or  lethality  testing  of  a  covered  system, 
major  munitions  program,  or  missile  program  before  the  program  can  proceed  beyond 
low-rate  initial  production.  Section  2366  of  title  10,  United  States  Code,  defines  a 
covered  system  as  any  vehicle,  weapon  platform,  or  conventional  weapon  system  that 
includes  features  designed  to  provide  some  degree  of  protection  to  users  in  combat  and 
that  is  a  major  system. 

Audit  Objectives.  The  primary  audit  objective  was  to  evaluate  the  DoD 
implementation  of  live-fire  testing  requirements  for  major  Defense  systems  in 
Section  2366  of  title  10,  United  States  Code.  We  also  evaluated  the  management 
control  program  as  it  related  to  our  audit  objective. 

Audit  Results.  Overall,  the  Office  of  the  Secretary  of  Defense  and  the  Military 
Departments  effectively  implemented  the  LFT&E  process  for  the  programs  reviewed. 
However,  four  areas  required  administrative  improvements  to  make  resources  available 
for  higher  priority  LFT&E  oversight  requirements. 

o  The  Office  of  the  Director,  Operational  Test  and  Evaluation  (DOT&E),  and 
the  Military  Departments  did  not  establish  folly  effective  procedures  to  correctly 
identify,  document,  and  place  all  LFT&E  candidate  systems  on  the  annual  test  and 
evaluation  oversight  list.  As  a  result,  DOT&E  did  not  have  assurance  that  the  annual 
test  and  evaluation  oversight  list  was  complete  and  that  the  Military  Departments 
performed  LFT&E  testing  for  those  systems  subject  to  LFT&E  requirements  that  were 
not  on  the  oversight  list  (Finding  A). 

o  The  DOT&E  could  further  streamline  the  LFT&E  reporting  process  resulting 
in  better  use  of  support  contractor  resources  to  perform  other  unfolfilled  LFT&E 
oversight  requirements  (Finding  B). 

o  For  3  of  13  major  systems  reviewed,  program  offices  did  not  process 
requests  for  waiver  from  foll-up,  system-level  survivability  or  lethality  tests  in  a  timely 
manner.  Further,  DoD  policy  for  processing  requests  for  waiver  from  foll-up,  system- 
level  survivability  or  lethality  tests  was  not  consistent.  Delays  in  submitting  and 
approving  a  request  for  waiver  increase  the  risk  that  program  offices  will  not  have 
sufficient  LFT&E  results  available  before  the  program  foil-rate  production  decision, 
causing  decisionmakers  to  postpone  the  foil-rate  production  decision,  the  Military 
Departments  to  slip  the  release  of  the  system  to  the  user,  and  the  program  costs  to 
increase  (Finding  C). 

o  The  LFT&E  Office  did  not  document  LFT&E  administrative  policies  and 
procedures  and  did  not  establish  procedures  for  maintaining  files  to  support  conclusions 
and  recommendations  made  on  individual  LFT&E  programs.  As  a  result,  LFT&E 
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action  officers  did  not  have  information  needed  to  perform  their  job  responsibilities 
most  efficiently,  did  not  maintain  consistent  and  complete  LFT&E  program  files,  and 
could  not  transfer  LFT&E  oversight  responsibilities  to  new  LFT&E  action  officers  in 
the  most  orderly  manner  (Finding  D). 

Recommendations  in  this  report,  if  implemented,  will  improve  the  DoD  live-fire  test 
and  evaluation  process  and  correct  the  material  management  control  weaknesses 
identified  in  the  report  (Appendix  A). 

Summary  of  Recommendations.  We  recommend  using  Military  Department 
Acquisition  Category  II  and  III  program  lists  to  help  identify  systems  subject  to 
LFT&E  requirements,  providing  a  draft  of  the  test  and  evaluation  oversight  list  to  the 
Military  Departments  for  review  and  comment,  and  eliminating  the  Institute  for 
Defense  Analyses  formal  LFT&E  technical  report  requirement.  We  also  recommend 
revising  DoD  guidance  to  establish  a  consistent  policy  for  submitting  requests  for 
waiver  from  full-up,  system-level  survivability  or  lethality  testing  and  finalizing  the 
"Director,  Operational  Test  and  Evaluation,  Notebook."  Further,  we  recommend 
formal  documentation  and  recommendation  of  LFT&E  candidate  systems.  We  also 
recommend  submitting  requests  for  waiver  from  full-up,  system-level  survivability  or 
lethality  testing  for  certain  systems.  Additionally,  we  recommend  placing  a  list  of 
Navy  Acquisition  Category  11  and  III  programs  on  the  Internet. 

Management  Comments.  The  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition)  and  the  Director,  Navy  Test  and  Evaluation  and 
Technology  Requirements,  did  not  respond  to  a  draft  of  this  report.  However,  we 
received  comments  on  a  draft  of  this  report  from  the  Director,  Operational  Test  and 
Evaluation;  the  Deputy  Under  Secretary  of  the  Army  (Operations  Research);  and  the 
Principal  Deputy,  Assistant  Secretary  of  the  Air  Force  (Acquisition).  The  Director, 
Operational  Test  and  Evaluation,  fully  or  partially  concurred  with  the  findings  and 
concurred  with  the  recommendations  or  provided  acceptable  alternative  corrective 
action.  The  Deputy  Under  Secretary  of  the  Army  concurred  with  the  findings  and 
recommendations.  The  Principal  Deputy  concurred  with  the  recommendations 
addressed  to  the  Air  Force  and  provided  comments  concerning  the  B-2  bomber. 
Although  not  required  to  comment,  the  Director,  Test,  Systems  Engineering  and 
Evaluation,  and  die  Director,  Air  Force  Test  and  Evaluation  Directorate,  provided 
comments  addressing  the  findings,  recommendations,  and  specific  aspects  of  the  report. 
See  Part  I  for  a  summary  of  management  comments  responding  to  the 
recommendations.  Part  II  for  our  response  to  the  management  comments  addressing 
specific  text  in  the  report,  and  Part  III  for  the  complete  text  of  management  comments. 

Audit  Response.  The  Director,  Operational  Test  and  Evaluation;  the  Deputy  Under 
Secretary  of  the  Army  (Operations  Research);  and  the  Principal  Deputy,  Assistant 
Secretary  of  the  Air  Force  (Acquisition),  comments  were  responsive  to  our 
recommendations.  Therefore,  no  additional  comments  are  required  from  those 
individuals  in  response  to  this  report.  However,  because  the  Assistant  Secretary  of  the 
Navy  (Research,  Development  and  Acquisition)  and  the  Director,  Navy  Test  and 
Evaluation  and  Technology  Requirements,  did  not  respond  to  a  draft  of  this  report,  we 
request  that  they  provide  comments  on  this  final  report  by  November  10,  1997. 
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Part  I  -  Audit  Results 


Audit  Background 


Section  2366  of  title  10,  United  States  Code,  "Major  systems  and  munitions 
programs:  survivability  and  lethality  testing  required  before  full-scale 

production, "  requires  the  Secretary  of  Defense  to  provide  that  a  covered  system, 
a  major  munitions  program,  or  a  product  improvement  to  a  covered  system  or 
major  munitions  program  not  proceed  beyond  low-rate  initial  production  until: 

o  the  completion  of  realistic  (full-up,^  system-level)  survivability  or 
lethality  testing  of  the  system  or  program,  as  applicable;  and 

o  the  Secretary  has  reported  the  results  of  such  testing  to  Congress 
along  with  the  Secretary's  overall  assessment  of  that  testing. 

Section  2366  of  title  10,  United  States  Code,  defines  a  covered  system  as  any 
vehicle,  weapon  platform,  or  conventional  weapon  system  that  includes  features 
designed  to  provide  some  degree  of  protection  to  users  in  combat  and  that  is  a 
major  system.  Realistic  survivability  testing  is  testing  for  vulnerability  of  a 
covered  system,  or  product  improvement  to  a  covered  system,  in  combat  by 
firing  munitions  likely  to  be  encountered  in  combat  (or  munitions  with  similar 
capability)  at  the  system  configured  for  combat,  with  the  primary  emphasis  on 
testing  vulnerability  with  respect  to  potential  user  casualties  and  taking  into 
equal  consideration  the  susceptibility  to  attack  and  combat  performance  of  the 
system.  Realistic  lethality  testing  is  testing  to  determine  the  lethality  of  a  major 
munitions  program,  missile  program,  or  a  covered  product  improvement 
program  by  firing  the  munition  or  missile  concerned  at  appropriate  targets 
configured  for  combat.  Appendix  C  provides  the  text  of  Section  2366  of 
title  10,  United  States  Code,  and  Subsection  2302(5)  of  title  10,  United  States 
Code,  that  Section  2366  references.  Appendix  D  provides  definitions  of 
technical  terms  used  in  this  report. 


Audit  Objectives 


The  primary  audit  objective  was  to  evaluate  the  effectiveness  of  the  DoD 
implementation  of  live-fire  testing  requirements  for  major  Defense  systems  in 
Section  2366  of  title  10,  United  States  Code.  Specifically,  we  evaluated 
whether  live-fire  tests  and  evaluations  provided  timely  and  reasonable 
assessments  of  the  survivability  and  lethality  of  covered  systems  as  they 
progressed  through  the  development  phase  before  the  full-rate  production 
decision.  Where  the  milestone  decision  authority  waived  the  use  of  full-up, 
system-level  survivability  or  lethality  tests,  we  evaluated  whether  the  milestone 
decision  authority  properly  made  the  determination  before  the  engineering  and 


^DoD  defines  a  full-up  live-fire  test  as  a  vulnerability  test  of  a  complete  or 
partial  system  loaded  or  equipped  with  all  dangerous  materials,  including 
flammables  and  explosives,  that  would  normally  be  on  board  in  combat. 
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manufacturing  development  decision  that  survivability  and  lethality  tests  of  a 
system  were  unreasonably  expensive  and  impractical  and  identified  acceptable 
alternative  survivability  or  lethality  tests.  We  also  evaluated  whether  the 
survivability  and  lethality  tests  were  adequately  planned,  conducted,  and 
reported  to  Congress  as  required.  Further,  we  evaluated  the  management 
control  program  as  it  related  to  our  audit  objective.  In  Appendix  A,  we  discuss 
the  scope  and  methodology  used  to  accomplish  the  objectives  as  well  as 
management  controls  and  another  matter  of  concern.  In  Appendix  B,  we 
summarize  prior  coverage. 


Live-Fire  Test  and  Evaluation  Generally  Well  Managed 


Overall,  the  Office  of  the  Secretary  of  Defense  and  the  Military  Departments 
effectively  implemented  the  LFT&E  process  for  the  programs  reviewed.  In 
fact,  after  the  Federal  Acquisition  Streamlining  Act  of  1994  transferred 
responsibility  for  monitoring  and  reviewing  live-fire  actions  from  the  Office  of 
the  Director,  Test,  Systems  Engineering  and  Evaluation,  to  the  Office  of  the 
Director,  Operational  Test  and  Evaluation  (DOT&E),  DOT&E  combined 
operational  test  and  evaluation  and  LFT&E  congressional  reporting 
requirements  into  a  single  report,  the  combined  beyond  low-rate  initial 
production  and  live-fire  report.  Combining  the  reports  has  made  for  a  more 
streamlined  and  effective  presentation  of  operational  test  and  LFT&E  results  to 
Congress  and  DoD  management. 

The  DOT&E  Live  Fire  Test  and  Evaluation  Office  has  also  employed  a  number 
of  creative  innovations  to  improve  awareness  of  the  LFT&E  process.  For 
example,  the  Office  has  written  a  number  of  papers,  conducted  LFT&E 
educational  conferences,  and  established  an  Internet  website^  to  inform 
individuals  involved  in  LFT&E  about  the  latest  LFT&E  advancements, 
guidance,  and  issues. 

Based  on  the  programs  reviewed  during  the  audit,  the  Military  Departments 
adequately  planned  and  conducted  survivability  and  lethality  tests  and  effectively 
executed  LFT&E  strategies  and  plans  with  appropriate  DOT&E  oversight.  For 
10  of  the  13  programs  reviewed,  the  Military  Departments  spent  less  than 
0.3  percent  of  a  program's  total  research,  development,  test,  and  evaluation  and 
procurement  funding  on  LFT&E. 

For  19  of  the  26  models  associated  with  the  programs  reviewed,  the  respective 
Military  Departments  were  planning  to  verify,  validate,  and  accredit  the  models 
before  using  them  for  live-fire  survivability  or  lethality  analyses.  However,  for 
the  seven  remaining  models,  the  Office  of  the  Secretary  of  Defense  and  the 
Military  Departments  did  not  plan  to  verify,  validate,  and  accredit  the  models. 
Nonetheless,  the  responsible  Military  Department  should  have  conducted 


^World-Wide  Website  [htq):// www.dote.osd.mil/lfte/INDEX. HTML]  contains 
information  on  the  DOT&E  live-fire  test  and  evaluation  program. 
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W&A  of  the  models.  Without  the  Military  Departments  conducting  the 
VV&A,  the  models  may  not  accurately  represent  the  developer's  conceptual 
description  and  specifications  and  may  not  be  an  accurate  representation  of  real- 
world  results  (Appendix  A).  Unrelated  to  the  VV&A  of  the  systems  reviewed, 
four  areas  required  administrative  improvements  to  make  resources  available  for 
higher  priority  LFT&E  oversight  requirements.  A  discussion  of  the  associated 
fmdings  follows. 
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Finding  A.  Live-Fire  Test  and 
Evaluation  Oversight  List 

The  DOT&E  and  the  Military  Departments  did  not  establish  fully 
effective  procedures  to  correctly  identify,  document,  and  place  all 
LFT&E  candidate  systems  on  the  annual  test  and  evaluation  oversight 
list.  The  DOT&E  and  Military  Department  procedures  were  not  fully 
effective  because: 

o  Military  Departments  did  not  have  documentation  showing  that 
they  recommended  systems  to  DOT&E  as  LFT&E  candidates, 

o  DOT&E  did  not  routinely  use  Military  Department  Acquisition 
Category  II  and  III  program  lists  to  identify  systems  subject  to  LFT&E 
requirements  in  preparing  the  annual  test  and  evaluation  oversight  list, 

0  DOT&E  did  not  provide  a  draft  of  the  test  and  evaluation 
oversight  list  to  the  Military  Departments  for  review  and  comment 
before  issuing  the  annual  test  and  evaluation  oversight  list,  and 

0  DoD  acquisition  policy  did  not  address  LFT&E  requirements 
for  systems  with  commercial  and  nontraditional  acquisition  strategies. 

As  a  result,  DOT&E  did  not  have  assurance  that  the  aimual  test  and 
evaluation  oversight  list  was  complete  and  that  the  Military  Departments 
performed  LFT&E  testing  for  those  systems  subject  to  LFT&E 
requirements  that  were  not  on  the  oversight  list. 


Live-Fire  Test  and  Evaluation  Requirements  and  Guidance 


Statutory  Requirements.  Section  139  of  title  10,  United  States  Code, 
"Director  of  Operational  Test  and  Evaluation,"^  and  Section  2366  of  title  10, 
United  States  Code,  establish  LFT&E  statutory  requirements. 

Section  139.  Section  139  requires  DOT&E  to  monitor  and  review  live- 
fire  testing  actions  in  DoD.  Section  139  also  requires  DOT&E  to  prepare  an 
annual  report  summarizing  the  operational  test  and  evaluation  actions,  including 
live-fire  testing,  of  DoD  during  the  preceding  fiscal  year.  Further,  Section  139 
requires  DOT&E  to  submit  concurrently  the  annual  report  to  the  Secretary  of 
Defense,  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  and 
Congress  not  later  than  10  days  after  the  President  submits  the  budget  to 
Congress  for  the  next  fiscal  year. 


^Public  Law  103-355,  "Federal  Acquisition  Streamlining  Act  of  1994," 
section  3012,  amended  section  139. 
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Finding  A.  Live-Fire  Test  and  Evaluation  Oversight  List 


Section  2366.  Section  2366  requires  the  Secretary  of  Defense  to  provide 
that  a  covered  system,  a  major  munitions  program,  or  a  product  improvement  to 
a  covered  system  or  major  munitions  program  not  proceed  beyond  low-rate 
initial  production  until: 

o  the  completion  of  realistic  (foll-up,  system-level)  survivability  or 
lethality  testing  of  the  system  or  program,  as  applicable;  and 

o  the  Secretary  has  reported  the  results  of  such  testing  to  Congress 
along  with  the  Secretary's  overall  assessment  of  that  testing. 

If  the  completion  of  realistic  survivability  or  lethality  testing  is  unreasonably 
expensive  and  impractical,  the  Secretary  may  waive  the  application  of  the 
survivability  and  lethality  tests  and  instead  allow  alternative  testing. 

DoD  Guidance.  The  DoD  Regulation  5000. 2-R,  "Mandatory  Procedures  for 
Major  Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated 
Information  System  (MATS)  Acquisition  Programs,"  March  15,  1996, 

Appendix  IV,  "Live  Fire  Test  and  Evaluation  Reports,  Mandatory  Procedures 
and  Formats,"  establishes  guidelines  for  conducting  LFT&E  within  DoD  and 
requires  the  Military  Departments  to  recommend  candidate  systems  for  LFT&E. 
Further,  the  regulation  requires  DOT&E  and  the  Director,  Test,  Systems 
Engineering  and  Evaluation,  to  jointly  publish  an  annual  test  and  evaluation 
oversight  list  of  programs  for  operational,  live-fire,  and  developmental  test  and 
evaluation  oversight. 

Army  Guidance.  Army  Regulation  73-1,  "Test  and  Evaluation  Policy," 
February  27,  1995,  requires  the  Deputy  Under  Secretary  of  the  Army 
(Operations  Research)  to  recommend  candidate  systems  for  LFT&E. 

Navy  Guidance.  Secretary  of  the  Navy  Instruction  5000.2B,  "Implementation 
of  Mandatory  Procedures  for  Major  and  Non-Major  Defense  Acquisition 
Programs  and  Major  and  Non-Major  Information  Technology  Acquisition 
Programs,"  December  6,  1996,  designates  the  Director,  Navy  Test  and 
Evaluation  and  Technology  Requirements,  as  the  Navy  LFT&E  primary  point 
of  contact.  Further,  the  instmction  requires  the  developing  organization  to 
implement  and  coordinate  LFT&E  requirements. 

Air  Force  Guidance.  Air  Force  Instruction  99-105,  "Live  Fire  Test  and 
Evaluation,"  July  25,  1994,  establishes  guidance  and  procedures  for  the  LFT&E 
of  Air  Force  systems.  The  instruction  requires  the  Assistant  Secretory  of  the 
Air  Force  (Acquisition)  to  recommend  candidate  systems  for  compliance  with 
LFT&E  legislation. 


Live-Fire  Test  and  Evaluation  Oversight  Candidates 


The  DoD  and  Military  Department  LFT&E  implementing  guidance  requires  the 
Military  Departments  to  recommend  candidate  systems  to  DOT&E  for  LFT&E; 
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however,  the  Military  Departments  did  not  have  documentation  showing  that 
they  recommended  systems  to  DOT&E  as  LFT&E  candidates. 

Army  Documentation.  The  Office  of  the  Deputy  Under  Secretary  of  the  Army 
(Operations  Research)  did  not  formally  document  and  recommend  candidates  for 
LFT&E  oversight.  Instead,  the  Office  informally  invited  representatives  from 
the  Office  of  the  DOT&E  to  attend  integrated  product  team  meetings  or 
program  briefings  on  those  programs  that  the  Army  believed  met  the  criteria  in 
DoD  Regulation  5000. 2-R  for  live-fire  oversight. 

Navy  Documentation.  The  Office  of  the  Director,  Navy  Test  and  Evaluation 
and  Technology  Requirements,  could  not  provide  documentation  showing  that 
developing  organizations  had  recommended  covered  systems  for  live-fire 
oversight.  A  representative  from  the  Office  of  the  Director  stated  that  he 
reviewed  Navy  acquisition  category  program  listings  to  identify  potential 
LFT&E  candidates.  If  a  program  appeared  to  meet  the  Section  2366  criteria 
and  was  not  on  the  DOT&E  live-fire  oversight  list,  he  then  counseled  the 
developing  organization  to  nominate  the  program  for  LFT&E.  However,  the 
representative  did  not  document  the  counseling. 

Air  Force  Documentation.  The  Office  of  the  Assistant  Secretary  of  the 
Air  Force  (Acquisition)  could  not  provide  documentation  showing  that  it  had 
recommended  systems  for  LFT&E  oversight.  The  Office  of  Sie  Assistant 
Secretary  stated  that  it  informally  identified  potential  live-fire  candidates  when  it 
reviewed  program  test  strategies  during  integrated  product  team  meetings. 
Operational  test  representatives  from  the  Office  of  the  DOT&E  were  normally 
present  at  those  meetings  if  the  program  was  subject  to  DOT&E  operational  test 
and  evaluation  oversight. 


Annual  Test  and  Evaluation  Oversight  List 


The  DOT&E  and  the  Military  Departments  did  not  establish  fully  effective 
procedures  to  correctly  identify,  document,  and  place  all  LFT&E  candidate 
systems  on  the  annual  test  and  evaluation  oversight  list.  Procedures  for 
establishing  and  maintaining  the  LFT&E  section  of  the  annual  test  and 
evaluation  oversight  list  were  not  fully  effective  because  DOT&E  did  not 
routinely  use  Military  Department  Acquisition  Category  II  and  III  program  lists 
to  identify  systems  subject  to  LFT&E  requirements  in  preparing  the  annual  test 
and  evaluation  oversight  list  and  did  not  provide  a  draft  of  the  test  and 
evaluation  oversight  list  to  the  Military  Departments  for  review  and  comment 
before  issuing  the  annual  test  and  evaluation  oversight  list.  DOT&E  could  use 
the  Military  Department  Acquisition  Category  n  and  III  program  lists  to 
identify  major  systems  and  munition  programs,  respectively,  for  inclusion  on  its 
oversight  list.  Further,  DoD  acquisition  policy  did  not  address  LFT&E 
requirements  for  systems  with  commercial  and  nontraditional  acquisition 
strategies. 
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Identifying  Live-Fire  Test  and  Evaluation  Oversight  Candidates.  Each 
year,  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology  publishes 
a  major  Defense  acquisition  programs  list  of  Acquisition  Category  I  programs. 
DOT&E  updates  the  annual  test  and  evaluation  oversight  list  by  using  the  list  of 
Acquisition  Category  I  programs  and  soliciting  feedback  from  operational  test, 
live-fire  test,  and  developmental  test  action  officers  in  the  Offices  of  the 
DOT&E  and  the  Director,  Test,  Systems  Engineering  and  Evaluation,  to 
determine  which  Acquisition  Category  II  and  III  programs  should  be  on  the 
aimual  test  and  evaluation  oversight  list.  DOT&E  action  officers  solicit  input 
from  their  respective  Military  Department  counterparts  for  selected  programs 
when  updating  the  list.  Based  on  fee  action  officer  feedback,  DOT&E  prepares 
a  tentative  list  of  test  and  evaluation  oversight  candidates.  Then,  DOT&E 
coordinates  fee  list  within  his  office  and  fee  Office  of  fee  Director,  Test, 
Systems  Engineering  and  Evaluation.  After  fee  coordination,  DOT&E  finalizes 
fee  list  and  both  DOT&E  and  fee  Director,  Test,  Systems  Engineering  and 
Evaluation,  sign  fee  list. 

Under  fee  annual  list  preparation  process,  DOT&E  cannot  be  certain  feat  he  has 
identified  all  Acquisition  Category  II  and  III  programs  feat  should  be  under 
LFT&E  oversight.  For  example,  DOT&E  did  not  identify  fee  M903  Caliber 
0.50  Saboted  Light  Armor  Penetrator  Ammunition  (M903)  and  fee  M430A1 
40  millimeter  (mm)  Cartridge  (M430A1)  as  LFT&E  candidates  on  its  annual 
test  and  evaluation  oversight  list.  Rather,  fee  Army  notified  DOT&E  after 
performing  a  review  of  its  munitions  programs  feat  feose  programs  should  be 
live-fire  candidates.  Later,  DOT&E  reported  on  feose  programs  to  Congress,  in 
accordance  wife  Section  2366. 

Further,  DOT&E  did  not  routinely  obtain  lists  of  all  Acquisition  Category  II 
and  III  programs  from  fee  Military  Departments  for  review  to  determine 
whether  fee  Militaiy  Departments  had  identified  all  LFT&E  candidate 
programs.  In  discussions  with  DOT&E  management,  they  agreed  feat  obtaining 
a  list  of  Military  Department  Acquisition  Category  II  and  III  programs  would  be 
helpful  in  identifying  LFT&E  candidate  programs.  The  Army'^  and  fee 
Air  Force^  already  maintain  program  acquisition  category  lists  on  fee  Internet. 
As  of  August  1997,  fee  Director,  Acquisition  Documentation,  Office  of  fee 
Assistant  Secretary  of  fee  Navy  (Research,  Development  and  Acquisition),  was 
in  fee  process  of  including  its  program  acquisition  category  list  on  fee  Internet. 
Because  fee  Military  Departments  have  made  or  are  mrfeing  program  acquisition 
category  lists  available  on  fee  Internet,  DOT&E  should  make  maximum  use  of 
feose  lists  to  identify  potential  LFT&E  candidates. 

Reviewing  and  Commenting  on  the  Test  and  Evaluation  Oversight 
List.  The  annual  test  and  evaluation  oversight  list  included  programs  feat  did 
not  belong  in  fee  LFT&E  section  of  fee  oversight  list,  according  to  fee 


^World-Wide  Website  [http://www.sarda.army.mil]  contains  Army  Acquisition 
Category  Listing. 

^World-Wide  Website  [http://afmc.wpafb.af.mil/organizations/HQ-AFMC/ 
DR/dri-home/pml.htm]  contains  Air  Force  Acquisition  Category  Listing. 
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Air  Force.  DOT&E  action  officers  informally  conferred  with  their  Military 
Department  counterparts  regarding  recommendations  for  test  and  evaluation 
oversight  candidates.  However,  DOT«&E  did  not  provide  the  subsequent  draft 
list  to  the  Military  Departments  for  review  and  comment  before  DOT&E  and 
the  Director,  Test,  Systems  Engineering  and  Evaluation,  issued  their  annual  test 
and  evaluation  oversight  list.  The  informal  process  at  the  action  officer  level 
could  not  ensure  that  DOT&E  and  the  Military  Departments  resolved 
differences  before  DOT&E  included  programs  on  the  oversight  list. 

Of  the  74  LFT&E  programs  listed  in  the  December  1996  annual  test  and 
evaluation  oversight  list,  DOT&E  annotated  4  as  being  "under  review  for  live 
fire  test. "  The  Office  of  the  DOT&E  included  those  programs  on  the  oversight 
list  to  maintain  visibility  pending  some  future  decision  regarding  the 
applicability  of  live-fire  testing.  In  February  1997,  the  Director,  Air  Force  Test 
and  Evaluation  Directorate,  identified  six  programs  that  he  believed  should  not 
be  in  the  LFT&E  section  of  the  oversight  list  and  proposed  to  DOT&E  the 
following  changes: 

o  remove  two  programs  from  LFT&E  oversight  because  the  LFT&E 
was  complete  and  DOT&E  had  issued  the  LFT&E  reports; 

o  annotate  another  program  as  "under  review  for  live-fire  test"; 

o  remove  one  program  from  the  list  because  it  was  already  in 

production; 

o  annotate  one  program  on  the  list  to  read  that  the  "requirements  of 

Title  10,  United  States  Code,  Section  2366,  would  apply  only  if  the  program 

proceeded  beyond  low-rate  [initial]  production";  and 

o  remove  one  program  from  the  list  because  its  system  survivability  was 
not  significantly  different  from  its  predecessor  system. 

Because  DOT&E  received  the  Director,  Air  Force  Test  and  Evaluation 

Directorate,  comments  after  DOT&E  published  the  annual  oversight  list,  the 
published  oversight  list  did  not  show  subsequent  changes  resulting  from 
discussions  between  DOT&E  and  the  Director,  Air  Force  Test  and  Evaluation 
Directorate,  concerning  those  comments. 

Addressing  Commercial  and  Nontraditional  Acquisition  Strategies. 
Section  2366  does  not  specifically  address  the  application  of  LFT&E 
requirements  for  commercial  acquisitions  or  acquisitions  that  do  not  follow  the 
milestone-based  acquisition  process.  Consequently,  the  Office  of  the  Secretary 
of  Defense  and  the  Military  Departments  have  the  flexibility  to  tailor  their 
conduct  and  oversight  of  the  LFT&E  process  for  coimnercial  and  nontraditional 
acquisitions  because  those  acquisitions  generally  do  not  follow  the  milestone- 
based  acquisition  process,  the  manufacturer  has  stabilized  the  product  design 
before  the  Office  of  the  Secretary  of  Defense  and  the  Military  Departments 
acquire  the  item,  and  the  item  may  already  be  in  production  before  DOT&E  and 
the  Military  Departments  have  identified  the  item  as  a  potential  live-fire 
candidate.  The  Office  of  the  DOT&E  and  the  Military  Departments  have 
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discussed  in  great  detail  whether  programs  with  commercial  and  nontraditional 
strategies  should  be  subject  to  LFT&E  oversight.  As  of  August  1997,  the 
Office  of  the  Secretary  of  Defense  was  considering  a  revision  to  DoD 
Regulation  5000. 2-R  to  require  commercial  items,  including  commercial  off- 
the-shelf  items,  nondevelopmental  items,  and  advanced  concept  technology 
demonstrators  or  prototypes  that  meet  the  definition  of  a  covered  system  or 
major  munitions  program  to  be  subject  to  LFT&E.  Therefore,  this  report 
makes  no  recommendation  to  address  the  issue. 


Effect  on  Managing  Live-Fire  Test  Oversight 


By  not  obtaining  and  reviewing  a  complete  list  of  all  Acquisition  Category  n 
and  in  programs,  DOT&E  did  not  have  assurance  that  all  programs  subject  to 
LFT&E  requirements  were  in  the  LFT&E  section  of  the  annual  test  and 
evaluation  oversight  list  and  that  Military  Departments  performed  appropriate 
LFT&E  for  tiiose  programs  that  were  not  on  the  oversight  list.  In  addition, 
because  the  Military  Departments  did  not  formally  document  and  recommend 
candidate  programs  for  LFT&E  oversight,  DOT&E  unnecessarily  spent  time 
obtaining  information  on  those  programs  from  other  unofficial  or  indirect 
sources.  Further,  by  not  providing  the  oversight  list  to  the  Military 
Departments  for  review  and  comment,  DOT&E  included  programs  that  may  not 
belong  on  the  oversight  list  that  DOT&E  submitted  to  the  Secretary  of  Defense, 
the  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  and  Congress 
for  program  oversight  purposes. 


Conclusion 


To  ensure  that  the  Office  of  the  DOT&E  oversees  all  acquisition  programs 
subject  to  LFT&E  requirements,  it  should  obtain  and  review  Military 
Department  acquisition  category  program  lists  when  preparing  the  annual  test 
and  evaluation  oversight  list.  The  routine  sharing  of  on-line  information  within 
the  DoD  acquisition  community  is  essential  for  enhanced  teamwork  and  for  less 
intrusive  oversight  envisioned  under  acquisition  reform.  To  further  enhance  the 
oversight  process,  DOT&E  should  provide  the  Military  Departments  with  the 
opportunity  to  formally  comment  on  programs  identified  for  oversight  and  to 
consider  Aose  comments  as  appropriate  before  issuing  the  annual  test  and 
evaluation  oversight  list,  and  the  Military  Departments  should  formally 
document  and  recommend  candidate  programs  for  LFT&E  oversight. 


Management  Comments  on  the  Finding  and  Audit  Response 


The  Director,  Operational  Test  and  Evaluation,  commented  on  the  finding.  The 
following  discussion  is  a  synopsis  of  those  comments  and  the  audit  response. 
The  Principal  Deputy,  Assistant  Secretary  of  the  Air  Force  (Acquisition),  and 
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the  Director,  Air  Force  Test  and  Evaluation  Directorate,  also  provided  specific 
comments  on  the  report.  See  Appendix  E  for  a  summary  of  those  comments 
and  the  audit  response.  The  complete  texts  of  those  comments  are  in  Part  III. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
partially  concurred  with  the  finding.  He  commented  on  the  maintenance  of  the 
aimual  test  and  evaluation  oversight  list  and  the  difficulty  associated  with 
determining  the  candidacy  of  systems  for  test  and  evaluation  oversight.  He 
stated  that  DoD  Regulation  5000. 2-R  requires  the  Military  Departments  to 
identify  potential  systems  for  LFT&E  oversight.  He  agreed  that  the 
identification  of  potential  LFT&E  systems  was  not  uniformly  implemented  with 
the  Military  Departments.  Further,  he  stated  that  DoD  Regulation  5000. 2-R 
explicitly  gives  DOT&E  the  responsibility  to  approve  LFT&E  candidate  systems 
for  oversight.  He  noted  that  the  identification  of  LFT&E  candidates,  especially 
for  Acquisition  Category  II  and  lower  category  programs,  has  not  been 
efficient.  Further,  he  noted  that  his  office  has  had  to  rely  on  any  number  of 
sources  to  obtain  information  to  identify  potential  candidates  when  the  Military 
Department  have  not  nominated  LFT&E  candidates  systems. 

Audit  Response.  We  agree  with  the  Director,  Operational  Test  and  Evaluation, 
comments. 


Recommendations,  Management  Comments,  and  Audit 
Response 


A.l.  We  reconunend  that  the  Assistant  Secretary  of  the  Air  Force 
(Acquisition);  the  Deputy  Under  Secretary  of  the  Army  (Operations 
Research);  the  Director,  Navy  Test  and  Evaluation  and  Technology 
Requirements,  formally  document  and  recommend  candidate  systems  to  the 
Director,  Operational  Test  and  Evaluation,  for  LFT&E  oversight. 

Deputy  Under  Secretary  of  the  Army  (Operations  Research) 
Comments.  The  Deputy  Under  Secretary  concurred  with  the  recommendation, 
stating  that  the  Army  would  join  in  whatever  procedure  that  DOT&E  established 
for  the  action. 

Principal  Deputy,  Assistant  Secretary  of  the  Air  Force  (Acquisition), 
Comments.  The  Principal  Deputy  concurred  with  the  recommendation,  stating 
that,  although  Air  Force  Instruction  99-105,  "Live  Fire  Test  and  Evaluation," 
July  25,  1994,  provides  guidance  and  procedures  for  the  LFT&E  of  Air  Force 
systems,  the  instruction  does  not  provide  specific  guidance  on  identifying 
candidate  systems.  Consequently,  the  Principal  Deputy  stated  that  the 
Air  Force  Test  and  Evaluation  Directorate  and  the  Office  of  the  Assistant 
Secretary  of  the  Air  Force  (Acquisition)  would  update  the  instruction  to  provide 
specific  guidance  concerning  the  systematic  documentation  and  recommendation 
of  Air  Force  candidate  systems.  The  Principal  Deputy  estimated  that  the  update 
would  be  completed  by  April  1998. 
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Director,  Navy  Test  and  Evaluation  and  Technology  Requirements, 
Comments.  The  Director  did  not  comment  on  the  recommendation.  We 
request  that  the  Director  provide  comments  in  response  to  the  final  report. 

Director,  Operational  Test  and  Evaluation,  Comments.  Although  not 
required  to  comment,  the  Director  agreed  that  the  Militaiy  Departments  should 
comply  with  current  DoD  regulations  requiring  the  Military  Departments  to 
nominate  potential  candidates  for  LFT&E  consideration. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 

A.2.  We  recommend  that  the  Director,  Operational  Test  and  Evaluation: 

a.  Use  Military  Department  Acquisition  Category  U  and  ID 
program  lists  to  help  identify  systems  subject  to  live-fire  test  and  evaluation 
requirements  in  preparing  the  annual  test  and  evaluation  oversight  list. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred  with  the  recommendation,  stating  that  the  Military  Departments 
should  provide  Acquisition  Category  II  and  III  program  lists  to  his  office  on  a 
regular  basis  for  review. 

Director,  Test,  Systems  Engineering  and  Evaluation,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director,  Air  Force  Test  and  Evaluation 
Directorate,  stated  that  he  did  not  believe  that  a  review  of  Acquisition 
Category  III  program  lists  would  help  identify  LFT&E  candidate  systems 
subject  to  Section  2366  of  title  10,  United  States  Code.  However,  he  did  agree 
with  the  use  of  Acquisition  Category  II  program  lists  to  identify  candidate 
systems.  He  stated  &at  Section  2366  describes  the  systems  that  are  subject  to 
LFT&E  and  specifically  addresses  Acquisition  Category  I  and  II  programs.  He 
further  stated  that,  according  to  the  definition  of  "covered  system"  in 
Subsection  2366(e)(1),  only  Acquisition  Category  I  and  II  programs  are 
covered.  He  stated  that  Acquisition  Category  III  programs  are  not  potential 
LFT&E  candidates  and  are  not  subject  to  Office  of  the  Secretary  of  Defense 
LFT&E  oversight. 

Audit  Response.  Acquisition  Category  HI  programs  could  be  subject  to 
LFT&E  oversight  when  they  include  major  munitions  programs  or  product 
improvements  to  a  covered  system  or  a  major  munitions  program  or  a  missile 
program  that  is  likely  to  affect  significantly  the  survivability  or  lethality  of  the 
applicable  system,  munition,  or  missile  produced  under  the  program. 

Subsection  2366(e)(2)  of  title  10,  United  States  Code,  states  that  a  major 
munitions  program  is  a  munitions  program  for  which  DoD  plans  to  acquire 
more  than  one  million  rounds  or  a  conventional  munitions  program  that  is  a 
major  system.  Consequently,  a  munitions  program  for  more  than  a  million 
rounds  of  munitions  would  probably  not  be  a  major  system;  however,  it  could 
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be  subject  to  LFT&E  oversight,  such  as  the  M903  Caliber  0.50  Saboted  Light 
Armor  Penetrator  Ammunition  (M903)  and  the  M430A1  40  mm  Cartridge 
(M430A1X  as  noted  in  this  finding. 

Subsection  2366(e)(6)  of  title  10,  United  States  Code,  states  that  a  covered 
product  improvement  program  is  a  modification  or  upgrade  to  a  covered  system 
or  to  a  major  munitions  program  or  a  missile  program  that  the  Secret^  of 
Defense  has  determined  will  likely  affect  significantly  the  survivability  or 
lethality,  respectively,  of  the  system  or  of  the  munition  or  missile  produced 
under  the  program.  Consequently,  an  Acquisition  Category  III  product 
improvement  to  a  covered  system  or  a  major  munitions  program  or  a  missile 
program  could  significantly  change  the  overall  survivability  or  lethality  of  the 
applicable  system,  munition,  or  missile  produced  under  the  program  and, 
therefore,  could  make  the  program  subject  to  LFT&E  oversight. 

b.  Provide  a  draft  of  the  test  and  evaluation  oversight  list  to  the 
Military  Departments  for  review  and  comment  before  issuing  the  annual 
test  and  evaluation  oversight  list. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred  with  the  recommendation,  stating  that  a  draft  of  the  test  and 
evaluation  oversight  list  should  be  circulated  to  the  Military  Departments  for 
their  review  and  comment  before  publication. 

Director,  Test,  Systems  Engineering  and  Evaluation,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  reconunendation. 

A.3.  We  recommend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition)  place  a  list  of  Navy  Acquisition  Category  II 
and  ni  programs  on  the  Internet  by  September  30,  1997. 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 
Comments.  The  Assistant  Secretary  did  not  comment  on  the  recommendation. 
We  request  that  the  Assistant  Secretary  provide  comments  in  response  to  the 
final  report. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 
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The  DOT&E  could  further  streamline  the  LFT&E  reporting  process. 
Before  the  transfer  of  LFT&E  oversight  responsibilities  to  DOT&E,  the 
Director,  Test,  Systems  Engineering  and  Evaluation,  reported  live-fire 
test  results  and  assessments  to  Congress  based  on  Institute  for  Defense 
Analyses  (IDA)^  technical  reports.  Since  the  transfer  of  LFT&E 
oversight  responsibilities,  DOT&E  includes  live-fire  test  results  and 
assessments  with  operational  test  results  in  a  DOT&E  combined  beyond 
low-rate  initial  production  and  live-fire  report  to  Congress.  However, 
DOT&E  live-fire  test  management  has  continued  to  require  IDA 
technical  reports  after  the  transfer  of  responsibilities.  The  DOT&E 
combined  report  that  includes  LFT&E  results  fulfills  DOT&E  live-fire 
and  operational  test  reporting  requirements.  As  a  result,  DOT&E  could 
better  use  IDA  resources  to  perform  other  unfulfilled  LFT&E  oversight 
requirements. 


Live-Fire  Test  Reporting  Requirements  and  Background 


Statutory  Reporting  Requirements.  Subsection  2366(d)  of  title  10,  United 
States  Code,  "Reporting  to  Congress,"  and  Subsection 2399(b)  of  title  10, 
United  States  Code,  "Operational  Test  and  Evaluation,"  establish  statutory 
requirements  for  the  reporting  of  live-fire  and  operational  test  results  to 
Congress. 

Subsection  2366(d).  Subsection  2366(d)  requires  that,  at  the  conclusion 
of  survivability  or  lethality  testing  for  covered  systems  and  programs,  the 
Secretary  of  Defense  will  submit  a  report  on  the  testing  to  the  congressional 
defense  committees.^  The  reports  are  to  describe  the  results  of  the  survivability 
or  lethality  testing  and  to  give  the  Secretary's  overall  assessment  of  the  testing 
for  each  system. 

Subsection  23990)).  Subsection  2399(b)  requires  DOT&E  to  analyze 
the  results  of  the  operational  tests  and  evaluations  that  a  Military  Department 
conducts  for  each  major  Defense  acquisition  program.  At  the  conclusion  of  the 
testing,  the  Director,  DOT&E,  will  prepare  a  report  stating  whether  the  test  and 


^The  Institute  for  Defense  Analyses  is  a  federally  funded  research  and 
development  center  that  DOT&E  contracts  with  for  technical  and  administrative 
support. 

^Subsection  2366(e)(7)  of  title  10,  United  States  Code,  provides  that  the 
congressional  defense  committees  are  the  Committee  on  Armed  Services  and  the 
Committee  on  Appropriations  of  the  Senate,  and  the  Committee  on  National 
Security  and  the  Committee  on  Appropriations  of  the  House  of  Representatives. 
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evaluation  performed  were  adequate  and  whether  the  results  of  such  test  and 
evaluation  confirm  that  the  items  or  components  actually  tested  are  effective  and 
suitable  for  combat.  Subsection  2399(b)  also  requires  DOT&E  to  submit  the 
report  to  the  Secretary  of  Defense,  &e  Under  Secretary  of  Defense  for 
Acquisition  and  Technology,  and  the  congressional  Defense  committees  before  a 
final  decision  to  proceed  with  a  program  beyond  low-rate  initial  production. 
The  Secretary  of  Defense  may  provide  comments  that  accompany  the  report  to 
Congress. 

Live-Fire  Test  Report  Streamlining.  The  Federal  Acquisition  Streamlining 
Act  of  1994  transferred  responsibility  for  monitoring  and  reviewing  LFT&E 
from  the  Director,  Test,  Systems  Engineering  and  Evaluation,  to  DOT&E. 
Before  the  transfer  of  responsibility,  the  Director,  Test,  Systems  Engineering 
and  Evaluation,  had  contracted  with  IDA  to  provide  technical  support  in 
executing  live-fire  test  oversight  and  assessment.  The  DOT&E  continues  to  use 
IDA  to  support  its  LFT&E  program.  Before  DOT&E  assumed  responsibility 
for  live-fire  test  oversight  in  January  1995,  the  Director,  Test,  Systems 
Engineering  and  Evaluation,  met  the  congressional  reporting  requirements  by 
using  IDA-prepared  LFT&E  technical  reports  as  the  basis  for  its  live-fire  test 
results.  However,  DOT&E  has  since  combined  operational  test  and  evaluation 
and  LFT&E  congressional  reporting  requirements  into  a  single  report,  the 
combined  beyond  low-rate  initial  production  and  live-fire  report. 

DoD  Reporting  Requirements.  The  DoD  Regulation  5000. 2-R,  Appendix  IV, 
requires  DOT&E  to  conduct  an  assessment  of  Military  Department  LFT&E 
programs  and  prepare  the  Secretary  of  Defense's  LFT&E  assessment  report  to 
Congress.  DOT&E  includes  the  LFT&E  assessment  with  operational  test 
results  in  the  beyond  low-rate  initial  production  report  if  the  program  is  also 
subject  to  operational  test  and  evaluation  oversight.  Otherwise,  DOT&E  issues 
a  separate  report  to  Congress  addressing  live-fire  test  results  and  its  assessment 
of  those  results.  Further,  the  regulation  requires  the  Military  Departments  to 
document  the  results  and  overall  evaluation  of  all  testing  identified  in  the 
LFT&E  strategy  for  LFT&E  programs  and  report  the  results  and  overall 
evaluation  to  DOT&E  no  later  than  120  days  after  the  Military  Department 
completes  the  testing.  The  Military  Department  report  is  to  facilitate  the 
DOT&E  independent  report  to  Congress.  Appendix  F  discusses  the  LFT&E 
reporting  and  oversight  process. 


Live-Fire  Test  and  Evaluation  Reporting 


The  DOT&E  could  further  streamline  the  LFT&E  reporting  process  resulting 
from  the  transfer  of  LFT&E  oversight  responsibilities  to  DOT&E.  The  Office 
of  the  DOT&E  has  continued  to  require  IDA  to  prepare  formal  technical  reports 
for  LFT&E  programs.  The  Office  should  have  discontinued  the  requirement 
after  the  Federal  Acquisition  Streamlining  Act  transferred  LFT&E  oversight 
responsibilities  to  DOT&E  from  the  Office  of  the  Director,  Test,  Systems 
Engineering  and  Evaluation,  and  after  DOT&E  began  preparing  its  combined 
beyond  low-rate  initial  production  and  live-fire  reports.  In  preparing  the 
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combined  beyond  low-rate  initial  production  and  live-fire  reports  as  discussed  in 
Appendix  F,  the  DOT&E  operational-test  action  officers  use  draft  summary 
input  on  the  live-fire  test  results  and  evaluation  from  IDA  rather  than  the  IDA- 
prepared  technical  report.  From  January  1995  through  May  1997,  DOT&E 
issued  two  beyond  low-rate  initial  production  reports  that  combined  operational 
test  results  with  live-fire  test  results  and  assessments.  Those  reports  were  the 
"AH-64D  Longbow  Apache  Attack  Helicopter,"  October  1995,  and  the  "C-17 
Globemaster  HI  Airlift  Aircraft,"  November  1995.  In  addition,  DOT&E  issued 
two  other  live-fire  test  reports  for  programs  subject  only  to  DOT&E  live-fire 
oversight.  Those  reports  were  the  "Live  Fire  Test  and  Evaluation  of  the  0.50 
Caliber  M903  Saboted  Light  Armor  Penetrator  (SLAP)  Ammunition," 
September  26,  1995,  and  the  "Live  Fire  Test  and  Evaluation  Report  on  the 
M430A1  40MM  Cartridge, "  May  30,  1996.  Appendix  G  provides  details  on 
the  four  reports. 

For  two  of  the  four  live-fire  test  reports,  IDA  did  not  prepare  a  technical  report 
until  after  DOT&E  transmitted  the  reports  to  Congress. 

o  For  the  C-17  Globemaster  III  airlift  aircraft,  an  IDA  technical  report 
was  not  available  before  DOT&E  transmitted  its  LFT&E  report  to  Congress. 
IDA  issued  a  final  technical  report  about  5  months  after  DOT&E  transmitted  its 
LFT&E  report  to  Congress. 

0  For  the  M430A1  munition  program,  an  IDA  technical  report  was  not 
available  before  DOT&E  transmitted  its  LFT&E  report  to  Congress.  IDA 
issued  its  final  technical  report  about  3  months  after  DOT&E  transmitted  its 
LFT&E  report  to  Congress. 

For  the  two  remaining  live-fire  test  reports  that  DOT&E  submitted  to  Congress, 
IDA  has  yet  to  submit  a  final  technical  report  to  DOT&E. 

0  For  the  AH-64D  Longbow  Apache  Attack  helicopter,  a  draft  IDA 
technical  report  was  available  for  the  Longbow  Apache  portion  of  the  AH-64D 
Program  before  DOT&E  transmitted  its  LFT&E  report  to  Congress  in  October 
1995.  IDA  had  not  finalized  its  technical  report  on  the  Longbow  Apache 
portion  of  the  AH-64D  Program  as  of  August  1997.  IDA  did  not  prepare  a 
technical  report  for  the  Longbow  Hellfire  modular  missile  portion  of  the 
AH-64D  Program  because  it  did  not  have  time  to  prepare  the  report. 

0  For  the  M903  munition  program,  IDA  did  not  prepare  a  final  LFT&E 
technical  report  because  it  did  not  have  time  to  prepare  the  report. 

As  evidenced,  DOT&E  action  officers  were  able  to  prepare  their  LFT&E 
reports  for  Congress  based  on  summary  input  from  IDA  without  needing  a 
formal  IDA  technical  report.  In  addition,  DOT&E  apparently  does  not  need  the 
formal  IDA  technical  report,  and  no  adverse  impact  resulted  from  not  having 
the  final  technical  report. 

Time  for  Preparing  Institute  for  Defense  Analyses  Reports.  The  IDA  did 
not  maintain  records  on  how  much  time  it  spent  on  writing  LFT&E  technical 
reports.  However,  IDA  employees  estimated  that  writing  and  staffing  a  paper 
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for  publication  could  take  from  6  weeks  to  3  months  in  addition  to  the  time 
required  to  do  the  analysis.  Because  the  IDA  level-of-effort  contract  with 
DOT&E  did  not  require  IDA  to  track  costs  incurred,  we  were  unable  to 
quantify  the  costs  of  preparing  IDA  technical  reports  for  LFT&E  programs. 

Reason  for  Preparing  Institute  for  Defense  Analyses  Technical 
Reports.  The  Office  of  the  DOT&E  stated  that  it  needed  a  formal  IDA 
technical  report  to  support  congressional  information  requirements.  However, 
as  discussed  earlier  in  ^s  report,  IDA  had  not  finalized  its  technical  report  until 
after  DOT&E  provided  its  report  to  Congress  and,  in  two  of  the  four  cases,  had 
not  prepared  the  formal  technical  report  more  than  IV2  years  later. 

The  Office  of  the  DOT&E  also  stated  that  the  Military  Departments'  LFT&E 
reports  could  not  support  the  DOT&E  report  to  Congress  because  the  Military 
Departments  might  not  be  as  candid  or  as  complete  in  reporting  unfavorable 
LFT&E  results  and  because  IDA  often  performed  DOT&E-directed  analyses 
that  the  Military  Departments  did  not  conduct  and  document  in  their  LFT&E 
reports.  In  our  opinion,  by  DOT&E  exercising  its  oversight  responsibilities  for 
approving  the  Military  Departments'  LFT&E  strategies,  monitoring  the  Militaiy 
Departments'  live-fire  tests,  and  reporting  the  live-fire  test  results  and  its 
assessment  of  those  results,  DOT&E  has  provided  sufficient  controls  over  the 
Military  Departments  to  ensure  that  the  Military  Departments  accurately  report 
LFT&E  results.  Although  IDA  often  performed  analyses  in  addition  to  those  of 
the  Military  Departments,  DOT&E  can  use  the  results  of  those  additional 
analyses  without  requiring  IDA  to  prepare  a  formal  technical  report. 


Effect  of  Live-Fire  Test  Reporting  Process 


By  DOT&E  not  further  streamlining  the  LFT&E  reporting  process,  DOT&E 
was  causing  IDA  to  spend  time  preparing  LFT&E  technical  reports,  time  which 
could  be  better  spent  performing  other  LFT&E  support  requirements.  Those 
LFT&E  support  requirements  could  include  overseeing  joint  live-fire  testing, 
providing  more  in-depth  oversight  to  covered  LFT&E  programs,  and  providing 
additional  support  to  LFT&E  educational  conferences.  The  technical  report 
preparation  time  could  be  significant  because  the  current  annual  test  and 
evaluation  oversight  list  has  74  live-fire  test  programs  that  would  benefit  from 
eliminating  the  formal  IDA  technical  report  requirement. 


Conclusion 


To  further  streamline  its  LFT&E  reporting  process,  DOT&E  should  eliminate 
the  requirement  for  the  IDA  formal  technical  report.  Instead  of  requiring  the 
formal  technical  report,  DOT&E  should  require  IDA  to  retain  its  technical 
analysis  documentation  necessary  to  support  its  draft  summary  input  on  life-fire 
test  results  and  evaluations  included  in  the  DOT&E  combined  beyond  low-rate 
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initial  production  and  live-fire  reports  to  Congress.  The  elimination  of  the 
formal  technical  report  and  the  retention  of  analyses  and  summary  information 
would  also  apply  to  live-fire  test  reports  for  programs  subject  only  to  DOT&E 
live-fire  oversight.  The  elimination  of  the  formal  technical  report  would  enable 
IDA  to  better  spend  its  time  performing  other  unfulfilled  DOT&E  live-fire  test 
and  evaluation  support  requirements. 


Management  Comments  on  the  Finding  and  Audit  Response 


The  Director,  Operational  Test  and  Evaluation;  the  Director,  Test,  Systems 
Engineering  and  Evaluation;  and  the  Director,  Air  Force  Test  and  Evaluation 
Directorate,  commented  on  the  finding.  The  following  discussion  is  a  synopsis 
of  the  Director,  Operational  Test  and  Evaluation,  and  the  Director,  Test, 
Systems  Engineering  and  Evaluation,  comments  and  the  audit  response.  The 
Director,  Air  Force  Test  and  Evaluation  Directorate,  comments  on  the  finding 
were  similar  to  his  comments  on  the  recommendation;  therefore,  we  discuss  his 
comments  on  the  finding  along  with  those  on  the  recommendation.  The 
complete  texts  of  the  comments  are  in  Part  III. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
partially  concurred  with  the  finding,  stating  that  statute  requires  him  to  prepare 
the  independent  live-fire  report  for  the  Secretary  of  Defense  and  the  four 
Defense-related  committees  of  Congress.  Therefore,  he  places  a  heavy 
emphasis  on  live-fire  report  preparation,  clarity,  completeness,  and  especially 
accuracy  and  timeliness.  Furdier,  he  discussed  the  assistance  that  IDA  provided 
in  the  preparation  of  the  live-fire  report.  He  felt  that,  with  the  move  of  the 
LFT&E  mission  to  the  DOT&E  office,  considering  LFT&E  results  in  the 
context  of  operational  test  and  evaluation  results  and  operational  test  and 
evaluation  results  in  the  context  of  LFT&E  results  made  a  lot  of  sense. 
Therefore,  for  those  programs  that  require  both  a  beyond  low-rate  initial 
production  report  and  live-fire  report,  he  stated  that  his  office  often  prepares 
and  submits  those  reports  under  one  cover.  He  stated  that  the  combination  of 
reports  enables  a  more  efficient,  balanced,  and  complete  assessment  of  the 
system.  However,  he  noted  that,  to  enable  the  combined  reports  to  be  more 
concise  and  readable,  his  office  removes  a  significant  amount  of  technical 
material  that  had  been  previously  included  in  LFT&E  reports.  He  added  that, 
nevertheless,  the  removed  material  is  critical  to  the  overall  conclusions  drawn  in 
the  combined  reports  and  should  be  retained  for  future  reference. 

Next,  the  Director  discussed  the  publishing  of  IDA  technical  reports.  He  noted 
that  the  draft  report  was  correct  in  stating  that  those  technical  reports  were  often 
published  long  after  the  live-fire  report  goes  to  Congress.  He  stated  that  the 
only  requirement  is  that  the  technical  work  be  performed  to  draw  the  correct 
conclusions  before  submission.  Further,  he  stated  that  the  technical  reports 
contain  vital  information  that  should  be  retained  for  future  use  and  reference. 
The  information  will  serve  as  a  baseline  for  comparison  with  future  upgrade 
systems  on  oversight.  As  a  result,  he  plans  to  maintain  those  files  for  future 
reference. 
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Audit  Response.  We  agree  with  the  Director,  Operational  Test  and  Evaluation, 
comments. 

Director,  Test,  Systems  Engineering  and  Evaluation,  Comments.  The 
Director  partially  agreed  with  the  finding,  stating  that  DOT«&E  should  have  the 
flexibility  to  direct  the  preparation  of  separate  technical  reports  when  the  reports 
appear  to  have  some  future  value.  The  preparation  of  separate  technical  reports 
is  pertinent  given  the  trend  to  use  past  test  data  to  reduce  the  costs  of  foture  test 
costs. 

Audit  Response.  DOT&E  should  have  the  flexibility  to  prepare  a  separate 
technical  report;  however,  DOT&E  should  only  prepare  the  formal  technical 
report  when  necessary.  As  the  Director,  Operational  Test  and  Evaluation, 
noted,  he  will  still  have  a  significant  amoimt  of  technical  material  available  for 
future  reference  even  if  IDA  does  not  prepare  a  formal  technical  report. 


Recommendation,  Management  Comments,  and  Audit 
Response 


B.  We  recommend  that  the  Director,  Operational  Test  and  Evaluation, 
eliminate  the  practice  of  obtaining  formal  live-Hre  test  and  evaluation 
technical  reports  from  the  Institute  for  Defense  Analyses  and,  instead, 
require  the  Institute  for  Defense  Analyses  to  retain  its  technical  analysis 
documentation  necessary  to  support  input  to  the  combined  beyond  low-rate 
initial  production  and  live-fire  reports  and  live-fire-oversight-only  reports 
to  Congress. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred  with  the  recommendation,  stating  that  he  has  removed  the 
requirement  from  the  renewal  contract  with  IDA  to  have  an  LFT&E  technical 
report  as  a  mandatory  deliverable  for  every  program. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  disagreed  with  the  finding  and 
recommendation.  The  Director  stated  that,  according  to  Subsection  2366(d)  of 
title  10,  United  States  Code,  the  Secretary  of  Defense,  not  DOT&E,  submits  the 
live-fire  report  to  Congress.  Further,  he  stated  that  independent  reporting  from 
DOT&E  is  only  required  for  initial  operational  test  and  evaluation  as  provided 
in  Section  2399  of  title  10,  United  States  Code,  but  is  not  required  in 
Section  2366  for  live-fire  reports.  He  stated  that,  by  combining  the  live-fire 
and  the  operational  test  and  evaluation  reports  and  submitting  both  of  those 
report  to  Congress  without  an  intervening  Secretary  of  Defense  review,  the 
Secretary's  control  over  the  live-fire  report  contents  before  submission  is 
compromised.  He  further  stated  that  the  beyond  low-rate  initial  production 
report  and  the  live-fire  report  must  remain  separate  to  ensure  that  the  Secretary 
of  Defense  can  edit  the  live-fire  report  before  the  Secretary  submits  the  report  to 
Congress,  as  mandated  by  Section  2366  of  title  10,  United  States  Code. 
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Audit  Response.  DoD  Directive  5000.1,  "Defense  Acquisition,"  March  15, 
1996,  which  the  Secretary  of  Defense  signed,  authorizes  the  publication  of  DoD 
Regulation  5000. 2-R.  The  DoD  Regulation  5000.2-R,  Appendix  IV,  requires 
DOT&E  to  conduct  an  assessment  of  Military  Department  LFT&E  programs 
and  prepare  the  Secretary  of  Defense's  LFT&E  assessment  report  to  Congress. 
Combining  the  beyond  low-rate  initial  production  (operational  test  and 
evaluation)  and  live-fire  reports  to  Congress  does  not  compromise  the  control  of 
the  Secretary  of  Defense  over  the  live-fire  report  contents  before  submission. 
Subsection  2399(b)  of  title  10,  United  States  Code,  requires  DOT&E  to  submit 
the  beyond  low-rate  initial  production  report  to  the  Secretary  of  Defense,  the 
Under  Secretary  of  Defense  for  Acquisition  and  Technology,  and  the 
congressional  Defense  committees  before  a  final  decision  to  proceed  with  a 
program  beyond  low-rate  initial  production.  Further,  the  subsection  states  that 
the  Secretary  of  Defense  may  provide  comments  that  accompany  the  report  to 
Congress.  Because  the  life-fire  report  is  combined  with  the  beyond  low-rate 
initial  production  report  to  the  Secretary,  the  Secretary  can  provide  comments 
that  accompany  the  report  to  Congress. 
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Waiver  From  Full-Up,  System-Level 
Survivability  or  Lethality  Tests 

For  3  of  13  major  systems  reviewed,  the  Military  Departments  did  not 
process  requests  for  waiver  from  full-up,  system-level  survivability  or 
lethality  tests  in  a  timely  manner.  The  Military  Departments  did  not 
process  the  requests  in  a  timely  manner  because  of  the  belief  that 
LFT&E  legislation  did  not  apply  to  one  of  the  systems,  because  of 
conflicting  LFT&E  guidance,  and  because  completing  the  waiver  process 
was  a  low  priority.  Unrelated  to  the  three  systems  reviewed,  the  DoD 
criteria  describing  the  waiver  process  were  not  consistent.  The  waiver- 
process  criteria  were  not  consistent  because  Section  3.4.9  and 
Appendix  IV  of  DoD  Regulation  5000.2-R  detail  different  procedures 
for  processing  a  request  for  waiver  from  full-up,  system-level 
survivability  or  lethality  testing.  Delays  in  submitting  and  approving  a 
request  for  waiver  increase  the  risk  that  program  offices  will  not  have 
sufficient  LFT&E  results  available  before  the  program  full-rate 
production  decision,  causing  decisionmakers  to  posq)one  the  full-rate 
production  decision,  the  Military  Departments  to  slip  the  release  of  the 
system  to  the  user,  and  the  program  costs  to  increase. 


Waiver  From  Full-Up,  System-Level  Survivability  or  Lethality 
Tests 


Statutory  Requirements.  Section  2366  of  title  10,  United  States  Code, 
provides  that  the  Secretary  of  Defense  may  waive  the  application  of 
survivability  or  lethality  tests  to  a  covered  system,  munitions  program,  or 
covered  product  improvement.  To  do  so,  the  Secretary  must  certify  to 
Congress,  before  the  system  or  program  enters  engineering  and  manufacturing 
development,  that  live-fire  testing  of  such  a  system  or  program  would  be 
unreasonably  expensive  and  impractical.  With  die  certification,  the  Secretary 
must  include  a  report  explaining  how  the  Secretary  plans  to  evaluate  the 
survivability  or  the  lethality  of  the  system  or  program  and  assessing  possible 
alternatives  to  realistic  (full-up,  system-level)  survivability  or  lethality  testing  of 
the  system  or  program. 

DoD  Requirements.  The  DoD  Regulation  5000.2-R,  Section  3.4.9,  "Live  Fire 
Test  and  Evaluation,"  and  Appendix  IV,  "Live  Fire  Test  and  Evaluation 
Reports,  Mandatory  Procedures  &  Formats,  Waivers,"  provide  guidance  for 
waivers  from  full-up,  system-level  survivability  or  lethality  testing. 

Section  3.4.9.  Section  3.4.9  states  that  the  Secretary  of  Defense 
delegated  authority  to  the  Defense  Acquisition  Executive  (DAE)  for  Acquisition 
Category  ID  programs  and  to  the  Component  Acquisition  Executive  (CAE)  for 
less  than  Acquisition  Category  ID  programs  to  waive  the  requirement  for 
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full-up,  system-level  survivability  or  lethality  tests  if  the  DAE  or  the  CAE 
certifies  to  Congress  before  the  program  enters  engineering  and  manufacturing 
development  that  full-up,  system-level  testing  of  the  program  would  be 
unreasonably  expensive  and  impractical. 

As  an  alternative  to  full-up,  system-level  survivability  or  lethality  tests,  the 
DAE  or  the  CAE  may  authorize  testing  of  a  system  or  program  by  fhing 
munitions  likely  to  be  encountered  in  combat  at  components,  subsystems,  and 
subassemblies,  together  with  performing  design  analyses,  modeling  and 
simulation,  and  analyses  of  combat  data.  Within  the  request  for  waiver  from 
full-UD,  system-level  survivability  and  lethality  testing,  the  respective  program 
office^  is  to  include  a  strategy  for  such  alternative  testing.  The  DOT&E  and  the 
Director,  Test,  Systems  Engineering  and  Evaluation,  are  to  jointly  review  the 
strategy,  and  DOT&E  is  to  approve  it.  Such  alternative  testing  may  not  be 
carried  out  unless  the  DAE  or  the  CAE  certifies  to  Congress  before  the  program 
enters  engineering  and  manufacturing  development  that  full-up,  system-level 
survivability  or  lethality  testing  of  the  program  would  be  unreasonably 
expensive  and  impracticable. 

The  respective  program  office®  is  to  address  waivers  and  the  use  of  alternative 
survivability  and  lethality  testing  in  the  test  and  evaluation  master  plan  for  the 
covered  system,  program,  or  covered  product  improvement  program.  Further, 
Section  3.4.9  states  that  the  respective  Military  Department  is  to  submit  CAE 
certifications  and  reports,  required  under  Subsection  2366(c)  of  title  10,  United 
States  Code,  to  Congress  through  DOT&E  and  the  DAE. 

Appendix  IV.  Appendix  IV  states  that  the  Military  Department 
requesting  a  waiver  from  foll-up,  system-level  survivability  or  lethality  testing 
will  include  a  report  explaining  how  the  Military  Department  plans  to  evaluate 
the  survivability  or  lethality  of  the  system  or  program  and  will  assess  possible 
alternatives  to  fiill-up,  system-level  survivability  or  lethality  testing.  Further, 
Appendix  IV  states  that  Ae  Military  Department  Secretary  is  to  submit  a  request 
for  waiver  from  full-up,  system-level  survivability  or  lethality  testing  to  the 
Deputy  Secretary  of  Defense  after  DOT&E  has  formally  coordinated  the  request 
for  waiver. 


Obtaining  Waivers  for  Three  Major  Systems 


For  3  of  13  major  systems  reviewed,  the  Military  Departments  did  not  process 
requests  for  waiver  from  full-up,  system-level  survivability  or  lethality  tests  in  a 
timely  manner.  The  three  major  systems  were:  the  Arleigh  Burke  (DDG-51) 


^Section  3.4.9  does  not  specify  who  prepares  the  request  for  waiver;  however, 
Section  3.4.9  implies  that  it  is  the  program  office. 

^Section  3.4.9  does  not  specify  who  submits  the  request  for  waiver;  however. 
Section  3.4.9  implies  that  it  is  the  Military  Department. 
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Class  Destroyer,  the  F-22  Aircraft,  and  the  MH-47E  and  MH-60K  Special 
Operations  Aircraft  (Special  Operations  Aircraft  Program).  The  Military 
Departments  did  not  process  the  requests  in  a  timely  manner  because  of  the 
belief  that  LFT&E  legislation  did  not  apply  to  the  DDG-51  ship  class,  because 
of  conflicting  LFT&E  guidance,  and  because  completing  the  waiver  process  was 
a  low  priority. 

DDG-51  Ship  Class  Request  for  Waiver.  The  Navy  nominated  the  entire 
DDG-51  ship  class  as  an  LFT&E  candidate  on  April  20,  1987.  On 
November  9,  1992,  the  Director,  Test  and  Evaluation  (now  the  Director,  Test, 
Systems  Engineering  and  Evaluation),^®  designated  Flight  IIA  of  the  DDG-51 
ship  class  as  a  major  product  improvement  program  subject  to  LFT&E. 
Flight  IIA  entered  Milestone  IV,  major  modification  approval,  on  February  2, 
1994.  For  the  entire  DDG-51  ship  class,  including  Flights  I,  11,  and  HA,  the 
DDG-51  Program  Office  submitted  a  request  for  waiver  from  full-up,  system- 
level  survivaWlity  testing  to  the  Office  of  the  Assistant  Secretary  of  the  Navy 
(Research,  Development  and  Acquisition)  (the  Assistant  Secretary)  on 
Au^st  28,  1995.  The  DDG-51  Program  Office  submitted  the  request  for 
waiver  because  full-up,  system-level  survivability  testing  of  the  entire  DDG-51 
ship  class  would  be  unreasonably  expensive  and  impractical. 

As  of  August  1997,  the  Office  of  the  Assistant  Secretary  had  not  yet  submitted 
to  Congress  the  request  for  waiver  from  ftill-up,  system-level  survivability 
testing  for  the  entire  DDG-51  ship  class  because  it  believed  that  Section  2366  of 
title  10,  United  States  Code,  did  not  apply  to  the  original  DDG-51  ship  class. 
The  Office  of  the  Assistant  Secretary  believed  that  Section  2366  did  not  apply 
because  the  milestone  decision  auAority  approved  the  DDG-51  ship  class 
proceeding  beyond  low-rate  initial  production  into  full-rate  production  in 
October  1986  and  because  it  believed  that  Section  2366: 

0  only  applied  to  decisions  made  after  May  31,  1987,  to  proceed 
beyond  low-rate  initial  production  and 

o  did  not  permit  requests  for  waiver  from  full-up,  system-level 
survivability  testing  after  a  milestone  decision  authority  approves  a  program 
entering  engineering  and  manufacturing  development,  Milestone  H. 

Even  though  the  Office  of  the  Assistant  Secretary  does  not  believe  that  it  needs 
to  submit  a  request  for  waiver  from  full-up,  system-level  survivability  testing, 
the  Navy  must  request  a  waiver  or  must  conduct  a  full-up,  system-level 
survivability  test  of  the  DDG-51  ship  class  because  the  system  is  subject  to 
LFT&E.  The  Office  of  the  DOT&E  has  continued  to  oversee  the  DDG-51  ship 
class  as  an  LFT&E  system  based  on  the  Navy  nomination  of  the  ship  class  as  an 
LFT&E  candidate  and  the  Director,  Test  and  Evaluation,  designation.  The 
Director,  Test  and  Evaluation,  designated  Flight  IIA  of  the  DDG-51  ship  class 
as  a  major  product  improvement  program  subject  to  LFT&E.  Further,  the 


l®The  Federal  Acquisition  Streamlining  Act  of  1994  transferred  responsibility 
for  monitoring  and  reviewing  LFT&E  from  the  Director,  Test,  Systems 
Engineering  and  Evaluation,  to  DOT&E. 
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DDG-51  Program  Office  is  implementing  an  alternative  LFT&E  strategy  based 
on  the  assumption  that  the  Office  of  the  Secretary  of  Defense  and  Congress  will 
approve  a  request  for  waiver  from  full-up,  system-level  survivability  testing.  If 
the  Office  of  the  Secretary  of  Defense  and  Congress  do  not  approve  the  request 
for  waiver  and  associated  alternative  LFT&E  strategy,  the  Navy  may  have  spent 
additional  resources  to  implement  the  alternative  LFT&E  strategy  and,  as  a 
consequence,  may  incur  additional  cost  and  may  slip  the  schedule. 

F-22  Aircraft  System  Request  for  Waiver.  In  April  1991,  the  F-22  System 
Program  Office  agreed  with  the  Office  of  the  Secretary  of  Defense  that  the  F-22 
Aircraft  system  was  a  covered  system  subject  to  LFT&E.  In  August  1991,  the 
F-22  Aircraft  system  entered  into  the  engineering  and  manufacturing 
development  phase  of  the  acquisition  process.  At  that  time,  the  F-22  System 
Program  Office  intended  to  perform  survivability  testing  on  materials, 
components,  and  subassemblies  for  the  F-22  Aircraft  system  instead  of  fiill-up, 
system-level  survivability  testing  based  on  LFT&E  guidance  that  the  Office  of 
the  Secretary  of  Defense  published  in  1988.  The  guidance  states  that  "Full-up 
testing  may  include  firings  against  full-scale  systems,  subsystems, 
subassemblies,  components,  or  acceptable  surrogates  that  allow  realistic 
threat/weapon  interaction  to  occur. " 

In  1992,  the  Office  of  the  Director,  Test  and  Evaluation,  notified  the  F-22 
System  Program  Office  that  the  1988  LFT&E  guidance  conflicted  with 
Section  2366  of  title  10,  United  States  Code.^^  Based  on  the  notification,  the 
F-22  System  Program  Office  began  working  with  the  Office  of  the  Secretary 
Defense  on  an  alternative  LFT&E  strategy  that  complied  with  Section  2366  and 
the  Air  Force  intention  to  perform  less  than  full-up,  system-level  survivability 
testing  for  the  F-22  Aircraft  system.  Consequently,  the  National  Defense 
Authorization  Act  for  FY  1995  directed  the  Secretary  of  Defense  to  request  the 
National  Research  Council  to  conduct  a  study  regarding  the  practicality  of  full- 
up,  system-level  survivability  testing  for  the  F-22  Aircraft  system.  The 
National  Research  Council  concluded  in  July  1995  that  the  F-22  System 
Program  Office  could  conduct  live-fire  testing  at  levels  below  a  full-up,  system- 
level  ground  test,  which  would  justify  a  request  for  waiver  from  full-up, 
system-level  survivability  testing  on  the  F-22  Aircraft  system  (Appendix  B). 

Based  on  the  National  Research  Council  Study,  Congress  enacted  Public  Law 
104-201,  Section  215,  "Live-Fire  Survivability  Testing  of  F-22  Aircraft," 
September  23,  1996.  The  law  states  that  the  Secretary  of  Defense  may,  in 
accordance  with  Subsection  2366(c)  of  title  10,  United  States  Code,  waive  for 
the  F-22  Aircraft  system  the  survivability  tests  that  the  section  requires,  even 
though  the  system  has  entered  engineering  and  manufacturing  development. 
The  F-22  System  Program  Office  still  needs  to  submit  the  request  for  waiver 
from  fiill-up,  system-level  survivability  testing  because  the  request  includes  the 
LFT&E  alternative  strategy  for  evaluating  the  survivability  of  the  F-22  Aircraft 
system  even  though  Congress  has  effectively  approved  the  request  for  waiver. 


l^In  January  1993,  the  National  Research  Council  issued  a  report  that 
concludes,  in  part,  that  the  1988  LFT&E  guidelines  and  the  1989  LFT&E 
planning  guide  were  not  consistent  with  the  live-fire  test  law  (Appendix  B). 
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Congressional  authorization  to  submit  the  request  for  waiver  from  full-up, 
system-level  survivability  testing  does  not  remove  the  requirement  for  die  F-22 
System  Program  Office  to  perform  survivability  testing  of  components, 
subsystems,  and  subassemblies. 

During  our  audit,  the  F-22  System  Program  Office  began  processing  its  request 
to  the  DAE,  through  the  CAE,  for  waiver  from  full-up,  system-level 
survivability  tests  of  fee  F-22  Aircraft  system.  As  of  August  4,  1997,  DOT&E 
approved  fee  alternative  LFT&E  plan  in  fee  test  and  evaluation  master  plan  for 
fee  F-22  Aircraft  system  and  recommended  that  fee  DAE  grant  fee  waiver.  The 
F-22  System  Program  Office  anticipates  feat  fee  waiver  will  be  granted  before 
fee  end  of  FY  1997.  By  having  an  approved  waiver,  fee  F-22  System  Program 
Office  will  have  sufficient  LFT&E  results  available  l^fore  fee  program  full-rate 
production  decision  and  fee  decisionmakers  will  not  have  to  postpone  fee 
full-rate  production  decision  until  LFT&E  results  are  available. 

Special  Operations  Aircraft  Program  Request  for  Waiver.  Initially,  fee 
Special  Operations  Aircraft  Program  (fee  Program)  was  not  a  major  system, 
subject  to  LFT&E.  However,  in  September  1991,  after  fee  Program  had  passed 
fee  low-rate  initial  production  phase  in  February  1990,  fee  Army  Audit  Agency 
concluded  that  fee  Program  had  breached  fee  Acquisition  Category  11  research 
and  development  funding  threshold  because  of  an  increase  in  research  and 
development  funds,  which  would  make  it  a  major  system.  Consequently,  fee 
Program  became  a  covered  system,  subject  to  live-fire  test  and  evaluation,  in 
October  1991.  However,  obtaining  a  waiver  from  full-up,  system-level 
survivability  testing  was  a  low  priority  for  fee  Special  Operations  Aircraft 
Program  Office.  In  October  1992,  Congress  enacted  Public  Law  102-484, 
Section  142,  which  required  fee  Special  Operations  Aircraft  Program  Office  to 
complete  survivability  testing  of  fee  Program  before  its  full-materiel  release 
date.  The  Special  Operations  Command  has  scheduled  full-materiel  release  for 
March  1998. 

The  Army  Program  Executive  Officer,  Aviation,  managed  fee  Special 
Operations  Aircraft  Program  until  April  1995,  at  which  time  fee  Special 
Operations  Command  took  over  management  of  fee  Program.  During  our 
audit,  fee  Special  Operations  Command  began  working  on  a  request  for  waiver 
from  full-up,  system-level  survivability  testing  for  fee  Program.  From 
December  1996  through  February  1997,  DOT&E  and  fee  Special  Operations 
Command  negotiated  a  revised  live-fire  strategy  feat  they  would  include  wife 
fee  request  for  waiver  from  full-up,  system-level  survivability  testing.  On 
June  4,  1997,  fee  Special  Operations  Command  Acquisition  Executive  signed 
and  submitted  to  Congress,  tluough  DOT&E,  fee  request  for  waiver,  including 
an  alternative  test  strategy.  Therefore,  this  report  rntJces  no  recommendation  to 
address  fee  issue. 


Process  for  Submitting  Requests  for  Waiver 


DoD  Regulation  5000. 2-R  is  not  consistent  within  itself  concerning  fee  process 
for  submitting  requests  for  waiver  from  full-up,  system-level  survivability  or 
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lethality  testing.  Depending  on  the  program  acquisition  category,  DoD 
Regulation  5050.2-R,  Section  3.4.9,  states  that  the  DAE  or  the  CAE  is 
responsible  for  waiving  the  requirements  for  full-up,  system-level  survivability 
or  lethality  testing  and  for  making  required  certifications  to  Congress.  Further, 
Section  3.4.9  states  that  DOT&E  and  the  Director,  Test,  Systems  Engineering 
and  Evaluation,  will  jointly  review  the  alternative  testing  strategy,  and  DOT&E 
will  approve  it. 

In  contrast,  DoD  Regulation  50(X).2-R,  Appendix  IV,  states  that  the  Military 
Department  Secretaries  will  submit  requests  for  waiver  from  full-up,  system- 
level  survivability  or  lethality  testing  to  the  Deputy  Secretary  of  Defense  (rather 
than  the  DAE  or  the  CAE)  after  DOT&E  has  formally  coordinated  the  request 
for  waiver.  Therefore,  Appendix  IV  implies  that  die  Deputy  Secretary  of 
Defense  is  responsible  for  waiving  the  requirements  for  full-up,  system-level 
survivability  or  lethality  testing  and  for  making  the  required  certifications  to 
Congress.  Consequently,  Appendix  IV  does  not  specifically  require  that  the 
DAE  or  the  CAE  be  involved  in  the  process  for  submitting  requests  for  waiver 
from  full-up,  system-level  survivability  or  lethality  testing. 


Conclusion 


Planning  and  conducting  of  LFT&E  cannot  be  accomplished  until  the  Office  of 
the  Secretary  of  Defense  determines  whether  a  system  will  be  subject  to  full-up, 
system-level  survivability  or  lethality  testing  or  to  alternative  survivability  or 
lethality  testing  of  components,  subsystems,  and  subassemblies,  together  with 
performing  design  analyses,  modeling  and  simulation,  and  analysis  of  combat 
data.  Test  organizations  need  to  know  information  on  a  system's  testing 
requirements  early  in  the  engineering  and  manufacturing  development  phase  of 
the  acquisition  process  to  ensure  that  the  Military  Department  plans  and 
performs  adequate  LFT&E  before  a  system's  full-rate  production  decision. 
Delays  in  submitting  and  approving  requests  for  waiver  from  full-up,  system- 
level  survivability  or  lethality  testing  increase  the  risk  that  program  offices  will 
not  have  sufficient  LFT&E  results  available  before  the  program  full-rate 
production  decision,  which  could  cause  decisionmakers  to  postpone  the  full-rate 
production  decision  and  the  Military  Departments  to  slip  the  release  of  systems 
to  the  user.  Also,  resources  will  be  wasted  if  the  Military  Departments  have  to 
redo  test  plans  or  tests.  Further,  DOT&E  needs  to  ensure  that  the  policy  for 
processing  and  submitting  requests  for  waiver  from  full-up,  system-level 
survivability  or  lethality  testing  is  consistent  in  Section  3.4.9  and  Appendix  IV 
of  DoD  Regulation  5000.2-R  to  preclude  potential  misunderstandings 
concerning  the  correct  way  to  process  requests  for  waiver  from  full-up,  system- 
level  survivability  or  lethality  testing.  To  ensure  consistent  waiver  process 
policy,  DOT&E  should  make  revision  recommendations  concerning  the  waiver 
process  in  DoD  Regulation  5000.2-R  to  the  Defense  Acquisition  Policy 
Working  Group,  the  group  responsible  for  changes  to  DoD 
Regulation  SOOO.^R. 
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Management  Comments  on  the  Finding  and  Audit  Response 


The  Director,  Operational  Test  and  Evaluation;  Director,  Test,  Systems 
Engineering  and  Evaluation;  and  the  Director,  Air  Force  Test  and  Evaluation 
Directorate,  commented  on  the  finding.  The  following  discussion  is  a  synopsis 
of  those  comments  and  the  audit  response.  The  complete  texts  of  the  comments 
are  in  Part  III. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred  with  the  finding,  stating  that  the  finding  was  certainly  appropriate  and 
that  his  office  has  been  dealing  with  the  issue  since  DoD  reissued  the  DoD  5000 
series  nearly  2  years  ago.  He  agreed  that  the  waiver  process  should  be  clarified 
and  made  consistent. 

Director,  Test,  Systems  Engineering  and  Evaluation,  Comments.  The 
Director  took  issue  with  the  audit  position  that  the  Office  of  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  and  Acquisition)  was  in  error  in 
believing  that  it  could  not  submit  a  request  for  waiver  from  full-up,  system-level 
survivability  testing  for  the  DDG-51  ship  class  after  the  program  passed 
Milestone  II,  Approval  to  Enter  Engineering  and  Manufacturing  Development. 
The  Director  opined  that  the  Office  of  the  Assistant  Secretary  may  be  correctly 
interpreting  Section  2366. 

Audit  Response.  We  continue  to  believe  that  the  Navy  is  not  correctly 
interpreting  Section  2366  of  title  10,  United  States  Code.  Congress  has  allowed 
DoD  to  submit  a  request  for  waiver  from  full-up,  system-level  survivability 
testing  after  a  system  has  entered  engineering  and  manufacturing  development. 
As  discussed  in  the  report.  Congress  enacted  Public  Law  104-201,  Section  215, 
"Live-Fire  Survivability  Testing  of  F-22  Aircraft,"  September  23,  1996,  stating 
that  the  Secretary  of  Defense  may,  in  accordance  with  Subsection  2366(c)  of 
title  10,  United  States  Code,  waive  for  the  F-22  Aircraft  system  the 
survivability  tests  that  the  section  requires,  even  though  the  system  has  entered 
engineering  and  manufacturing  development. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  The 
Director  stated  that  the  draft  report  presumes  that  the  DDG-51  ship  class  is 
subject  to  LFT&E  despite  the  Office  of  the  Assistant  Secretary  of  the  Navy 
(Research,  Development  and  Acquisition)  determination  that  the  system  was  not 
a  covered  system.  The  Director  stated  that  the  report  should  be  rephrased  to 
make  clear  diat  this  is  the  opinion  of  the  Inspector  General,  DoD.  The  Director 
rationalized  that  senior  DoD  leaders  are  currently  debating  the  issue  of  which 
systems  are  covered  systems  and  that  the  issue  could  impact  other  systems.  He 
stated  that  DoD  Regulation  5000.2-R,  Appendix  IV,  concedes  the  difficulty  of 
resolving  the  issue  when  the  regulation  states,  "Legal  counsel  should  be 
consulted  to  verify  the  final  determination  of  program  status."  He  summarized 
by  stating  that  generalizations  about  the  covered  status  of  systems  is  misleading. 

Audit  Response.  The  joint  DOT&E  and  Director,  Test,  Systems  Engineering 
and  Evaluation,  annual  test  and  evaluation  oversight  list  that  summarizes 
operational  test  and  evaluation  actions,  including  live-fire  testing,  designates  the 
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DDG-51  ship  class  (all  variants)  as  an  acquisition  program  subject  to  LFT&E 
oversight  as  well  as  operational  and  developmental  test  and  evaluation  oversight. 
DoD  Regulation  5000. 2-R  requires  DOT&E  and  the  Director,  Test,  Systems 
Engineering  and  Evaluation,  to  jointly  publish  an  annual  test  and  evaluation 
oversight  list  of  programs  for  operational,  live-fire,  and  developmental  test  and 
evaluation  oversight.  The  regulation  also  requires  DOT&E  to  approve 
candidate  systems  for  LFT&E  and  to  annually  review  all  potential  systems  for 
inclusion  or  exclusion  from  the  LFT&E  oversight  list. 

Concerning  DoD  Regulation  5000.2-R,  Appendix  IV,  and  the  statement  that 
"Legal  counsel  should  be  consulted  to  verify  the  final  determination  of  program 
status,"  the  regulation  also  states,  "If  any  doubt  exists,  the  system  should  be 
assumed  to  be  covered  and  appropriate  action  taken.  This  includes  waiver 
action  if  the  testing  would  be  unreasonably  expensive  and  impractical. " 


Recommendations,  Management  Comments,  and  Audit 
Response 


C.l.  We  recommend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition)  submit  to  Congress  the  request  for  waiver 
from  full-up,  system-level  survivability  testing  for  the  Arleigh  Burke 
(DDG-51)  Class  Destroyer  by  September  30,  1997. 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 
Comments.  The  Assistant  Secretary  did  not  comment  on  the  recommendation. 
We  request  that  the  Assistant  Secretary  provide  comments  in  response  to  the 
final  report. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director,  Air  Force  Test  and  Evaluation 
Directorate,  disagreed  with  the  recommendation,  stating  that  unless  the  DDG-51 
destroyer  is  identified  as  a  covered  system,  a  waiver  to  the  statute  is  not 
required. 

Audit  Response.  In  accordance  with  DoD  Regulation  5000.2-R,  DOT&E  has 
designated  the  DDG-51  ship  class  (all  variants)  as  an  acquisition  program 
subject  to  LFT&E  oversight  in  the  annual  test  and  evaluation  oversight  list. 
Therefore,  the  Navy  must  submit  a  request  for  waiver  from  full-up,  system- 
level  survivability  testing  or  must  conduct  a  full-up,  system-level  survivability 
test  of  the  DDG-51  ship  class. 

C.2.  We  recommend  that  the  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  submit  to  Congress,  through  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology,  the  request  for  waiver  from  full-up, 
system-level  survivability  testing  for  the  F-22  Aircraft  system  by 
September  30, 1997. 
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Survivability  or  Lethality  Tests 


Principal  Deputy,  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
Comments.  The  Principal  Deputy  concurred  with  the  recommendation,  stating 
that  his  office  intends  to  submit  the  request  for  waiver  from  full-up,  system- 
level  survivability  testing  for  the  F-22  Aircraft  system  by  September  30,  1997. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 

C.3.  We  recommend  that  the  Director,  Operational  Test  and  Evaluation, 
recommend  to  the  Defense  Acquisition  Policy  Working  Group  revisions  to 
Section  3.4.9  and  Appendix  IV  of  DoD  Regulation  5000.2-R,  "Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS)  Acquisition  Programs,"  March  15, 
1996,  to  establish  a  consistent  policy  for  submitting  requests  for  waiver 
from  full-up,  system-level  survivability  and  lethality  testing. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred,  stating  that  his  office  has  recently  published  portions  of  the 
"DOT&E  Notebook"  that  include  fully  explained  and  consistent  procedures  for 
processing  requests  for  waiver  from  fiill-up,  system-level  survivability  or 
lethality  testing.  Further,  he  plans  to  submit  to  the  Defense  Acquisition  Policy 
Working  Group  a  revised  regulation  for  consideration  and  inclusion  in  DoD 
Regulation  5000.2-R. 

Director,  Test,  Systems  Engineering  and  Evaluation,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 
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Controls 

The  LFT&E  Office  did  not  document  LFT&E  administrative  policies 
and  procedures  and  did  not  establish  procedures  for  maintaining  files  to 
support  conclusions  and  recommendations  made  on  individual  LFT&E 
programs.  The  LFT&E  Office  did  not  document  or  establish  procedures 
because: 

0  DOT&E  had  not  finalized  a  "DOT&E  Notebook"  that  contains 
LFT&E  administrative  policy  and  procedures  and  relied  on  a  contractor 
to  provide  file  documentation  and  maintenance  to  support  management 
decisions,  and 

0  the  LFT&E  Office  did  not  specifically  address  documenting 
LFT&E  administrative  policy  and  procedures  for  LFT&E  program 
oversight  in  its  1995  vulnerability  risk  assessment  associated  wi&  its 
management  control  program. 

As  a  result,  LFT&E  action  officers  did  not  have  information  needed  to 
perform  their  job  responsibilities  most  efficiently,  did  not  maintain 
consistent  and  complete  LFT&E  program  files,  and  could  not  transfer 
LFT&E  oversight  responsibilities  to  new  LFT&E  action  officers  in  the 
most  orderly  manner. 


Management  Control  Program  Background 


Management  Control  Program  Requirement*  The  DoD  Directive  5010.38, 
"Internal  Management  Control  Program,"  April  14,  1987,^^  requires  DoD 
managers  to  implement  a  comprehensive  system  of  management  controls  that 
provide  reasonable  assurance  that  programs  and  administrative  functions  are 
effectively  and  efficiently  carried  out  in  accordance  with  applicable  law  and 
management  policy  and  to  evaluate  the  adequacy  of  the  controls.  Management 
controls  are  the  organization,  policies,  and  procedures  that  managers  establish  to 
ensure  quality  and  timeliness  of  their  programs'  performance,  to  increase 
productivity,  to  control  costs,  to  mitigate  adverse  aspects  of  agency  operations, 
and  to  ensure  that  their  staff  manages  the  program  with  integrity  and  compliance 
with  applicable  laws. 


l^The  Office  of  DOT&E  used  the  April  1987  version  of  the  Directive  to 
perform  its  vulnerability  risk  assessment  for  the  LFT&E  function.  The  DoD 
revised  DoD  Directive  5010.38  as  "Management  Control  (MC)  Program," 
August  26,  1996. 
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Contractor  Study.  In  July  1996,  the  Deputy  Director,  Live  Fire  Test,  Office 
of  the  DOT«&E  (the  Deputy  Director),  initiated  a  study  with  International 
Development  and  Resources,  Incorporated  (the  Contractor),  to  evaluate  the 
LFT&E  processes.  The  purpose  of  the  study  was  to  propose  an  approach  for 
raising  the  productivity  of  LFT&E  personnel  in  the  Office  of  the  DOT&E  by 
process  improvement,  training,  and  the  application  of  technology.  For  the 
study,  the  Contractor  issued  to  the  Deputy  Director  a  final  report,  "Internal 
Automation  Support  for  LFT&E,"  May  5,  1997,  which  identified,  among  other 
things,  the  need  for  a  handbook  for  rotational  training  assignment  action 
officers.  lihe  handbook  would  contain  LFT&E  policies  and  office  procedures 
to  help  rotational  employees  transition  to  their  LFT&E  duties.  The  report  also 
identified  the  need  for  a  uniform  document  filing  system  and  a  central  computer 
data  base  for  LFT&E  program  documents  that  LFT&E  action  officers  could 
access  remotely.  The  report  provided  time  estimates  to  develop  and  implement 
the  uniform  document  filing  system  and  the  central  computer  data  base.  During 
our  audit,  die  Deputy  Director  requested  the  Contractor  to  expedite  completion 
of  a  draft  of  the  "Rotational  Training  Assignment  Action  Officer  Handbook"  for 
his  review.  As  of  August  1997,  DOT&E  officials  had  not  yet  received  the  draft 
handbook. 


Live-Fire  Test  and  Evaluation  Management  Controls 


The  LFT&E  Office  did  not  document  LFT&E  administrative  policies  and 
procedures  and  did  not  establish  procedures  for  maintaining  files  to  support 
conclusions  and  recommendations  made  on  individual  LFT&E  programs.  The 
LFT&E  Office  did  not  document  or  establish  procedures  because  DOT&E  had 
not  finalized  a  "DOT&E  Notebook"  that  contains  operational  and  live-fire  test 
and  evaluation  administrative  policy  and  procedures  and  relied  on  a  contractor 
to  provide  file  documentation  and  maintenance  to  support  conclusions  and 
recommendations  that  DOT&E  officials  made  on  individual  LFT&E  programs. 
In  addition,  the  LFT&E  Office  did  not  specifically  address  documenting 
LFT&E  administrative  policy  and  procedures  for  LFT&E  program  oversight  in 
its  1995  vulnerability  risk  assessment  associated  with  its  management  control 
program. 

Administrative  Policies  and  Procedures.  The  Office  of  the  DOT&E  had 
started  but  did  not  complete  an  effort  to  document  its  administrative  policies  and 
procedures,  including  LFT&E  administrative  policies  and  procedures.  Around 
February  1996,  DOT&E  initiated  the  development  of  a  "DOT&E  Notebook"  to 
provide  general  guidance  for  DOT&E  action  officers  and  to  supplement  the 
policy  and  procedures  for  operational  and  live-fire  tests  and  evaluations 
contained  in  die  DoD  Regulation  5000.2-R.  The  "DOT&E  Notebook"  contains 
information  on  the  DOT&E  organization,  testing  methodologies,  test  plans,  and 


l^Rotational  training  assignments  in  the  DOT&E  Live  Fire  Test  and  Evaluation 
Office  are  typically  for  1  year. 
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reports  to  Congress.  However,  as  of  August  1997,  the  "DOT&E  Notebook"  is 
still  in  draft.  DOT&E  anticipates  that  the  "DOT&E  Notebook"  will  be 
published  before  September  30,  1997. 

Maintaining  Files.  The  LFT&E  Office  did  not  have  a  standard  operating 
procedure  for  maintaining  program  documentation  such  as  test  and  evaluation 
master  plans,  operational  requirement  documents,  minutes  of  meetings,  and  trip 
reports  in  the  LFT&E  fdes.  The  Institute  for  Defense  Analyses  tracks  the  status 
of  each  LFT&E  program  for  the  LFT&E  Office.  When  auditors  requested 
program  documents  during  the  audit,  the  LFT&E  action  officers  generally  could 
not  locate  documents  because  they  did  not  have  a  standard  filing  system. 
Instead,  they  requested  the  documents  from  the  Institute  for  Defense  Analyses. 

Vulnerability  Risk  Assessment.  On  June  23,  1995,  the  Deputy  Director 
prepared  a  vulnerability  risk  assessment  in  accordance  with  DoD  Directive 
5010.38.  The  Deputy  Director  assessed  the  LFT&E  function  as  having  a  low 
risk  of  susceptibility  to  fraud,  waste,  or  mismanagement.  The  Deputy  Director 
did  not  specifically  address  the  lack  of  documented  LFT&E  Office 
administrative  policy  and  procedures  or  the  LFT&E  file  system  used  to  maintain 
documentation  to  support  conclusions  and  recommendations  that  DOT&E 
officials  made  on  individual  LFT&E  programs  for  LFT&E  program  oversight  in 
the  LFT&E  Office's  vulnerability  risk  assessment  associated  witih  its 
management  control  program. 


Effect  on  Management  Controls 


Without  a  finalized  "DOT&E  Notebook"  that  contains  LFT&E  administrative 
policy  and  procedures  and  that  establishes  procedures  for  maintaining  files, 
LFT&E  action  officers  did  not  have  information  needed  to  perform  their  job 
responsibilities  most  efficiently,  did  not  maintain  consistent  and  complete 
LFT&E  program  files,  and  could  not  transfer  LFT&E  oversight  responsibilities 
to  new  LFT&E  action  officers  in  the  most  orderly  manner. 


Conclusion 


The  Office  of  the  DOT&E  needs  to  complete  its  ongoing  action  to  document 
LFT&E  administrative  policies  and  procedures  in  Ae  "DOT&E  Notebook." 
The  Office  of  the  DOT&E  also  needs  to  establish  a  standard  filing  system  to 
ensure  compliance  with  management  policies  and  procedures  and  to  enable 
action  officers  to  perform  their  job  responsibilities  most  efficiently.  The 
establishment  of  a  standard  filing  system  will  improve  productivity  of  action 
officers  by  eliminating  the  need  for  action  officers  to  search  for  critical  program 
documents;  will  ensure  a  written  record  of  program  information,  such  as 
correspondence,  minutes  of  meetings,  and  analyses,  for  providing  program 
oversight;  and  will  reduce  the  time  that  a  new  action  officer  needs  to  become 
familiar  with  the  specifics  of  an  LFT&E  program. 
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Written  policies  and  procedures  are  essential  internal  controls  and  valuable  tools 
to  train  new  employees,  especially  in  the  LFT&E  Office  because  the  staff  of 
action  officers  regularly  includes  rotational  employees.  For  example,  as  of 
January  1,  1997,  the  Deputy  Director  had  a  staff  of  seven  action  officers, 
including  three  rotational  employees.  Two  of  the  three  rotational  employees 
were  on  board  fewer  than  3  months,  and  three  of  the  four  permanent  LFT&E 
action  officers  had  been  on  board  for  less  than  a  year. 

To  help  new  action  officers  transition  to  LFT&E  responsibilities,  the  Deputy 
Director  has  begun  developing  the  "Rotational  Training  Assignment  Action 
Officer  Handbook."  In  our  opinion,  DOT&E  needs  to  finalize  and  issue  the 
"DOT&E  Notebook"  and  needs  to  include  appropriate  information  from  the 
"Rotational  Training  Assignment  Action  Officer  Handbook"  in  the  "DOT&E 
Notebook. " 


Management  Comments  on  the  Finding  and  Audit  Response 


Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
partially  concurred  with  the  finding,  noting  that  the  DoD  5000  series  has  more 
descriptive  material  on  live-fire  testing  than  for  nearly  any  other  subject.  He 
stated  that  his  office  uses  the  creative  innovations  and  the  DoD  5000  series  to 
help  train  the  LFT&E  staff,  including  the  one-year  rotational  trainees  in  his 
office.  Therefore,  the  Director  stated  that  he  could  only  partially  concur  with 
the  finding  that  LFT&E  administrative  policies  and  procedures  were  not  well 
known  and  established.  The  complete  text  of  the  comments  are  in  Part  HI. 

Audit  Response.  In  view  of  the  agreed  upon  actions  in  response  to  the  finding, 
the  extent  to  which  the  policies  and  procedures  are  not  well  known  and 
established  in  the  absence  of  documentation,  is  a  moot  issue. 


Recommendations,  Management  Comments,  and  Audit 
Response 


D.  We  recommend  that  the  Director,  Operational  Test  and  Evaluation: 

1.  Complete  and  issue  by  September  30,  1997,  the  "Director, 
Operational  Test  and  Evaluation,  Notebook"  for  action  officers  to  include 
written  policies  and  procedures  to  describe  and  document  the  live-flre  test 
and  evaluation  management  control  process,  appropriate  information  from 
the  "Rotational  Training  Assignment  Action  Officer  Handbook,"  and  a 
standard  system  for  maintaining  live-fire  test  and  evaluation  program  fries. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred,  stating  that  his  office  has  accelerated  the  completion  of  the  "DOT&E 
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Notebook."  He  further  stated  that  the  full  integration  of  the  LFT&E  policies 
and  procedures  has  been  completed  in  draft  and  should  be  published  before 
September  30,  1997. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 

2.  Review  Live-Fire  Test  and  Evaluation  Office  administrative 
policy  and  procedures  and  the  related  system  used  to  document  and 
support  conclusions  and  recommendations  made  for  live-fire  test  and 
ev^uation  program  oversight  as  part  of  its  management  control  program 
self-evaluation.  Appendix  A  provides  details  on  the  adequacy  of  the 
management  control  program  self-evaluation. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred,  stating  that  his  office  will  review  LFT&E  administrative  procedures 
relating  to  the  documentation  and  support  of  conclusions  and  recommendations 
made  for  LFT&E  program  oversight. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  Although 
not  required  to  comment,  the  Director  agreed  with  the  recommendation. 


Part  n  -  Additional  Information 
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Scope  and  Methodology 


We  conducted  this  audit  from  August  1996  through  June  1997  and  reviewed 
documentation  dated  from  September  1986  through  May  1997.  To  accomplish 
the  audit  objective,  we: 

o  discussed  LFT&E  policy  and  procedures  with  personnel  in  the  Office 
of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  the  Office 
of  the  DOT&E,  and  selected  Military  Department  program  offices,  test 
organizations,  and  contractor  facilities; 

o  assessed  whether  DOT&E  and  the  Military  Departments  were 
performing  LFT&E  analyses  in  accordance  with  Section  2366  of  title  10,  United 
States  Code,  and  DoD  Regulation  5000. 2-R; 

o  reviewed  the  DOT&E  live-fire  test  and  evaluation  reporting  process 
for  reports  provided  to  DoD  milestone  decision  authorities  and  Congress; 

o  reviewed  requests  for  waiver  from  full-up,  system-level  survivability 
and  lethality  testing  to  determine  whether  the  milestone  decision  authority  made 
the  determination  before  the  engineering  and  manufacturing  development 
decision  that  survivability  or  lethality  testing  of  a  system  was  unreasonably 
expensive  and  impractical  and  identified  acceptable  alternative  survivability  or 
lediality  tests; 

0  assessed  whether  the  Militaty  Departments  conducted  verification, 
validation,  and  accreditation  of  modeling  and  simulation  applications  that  the 
Military  Departments  used  for  survivability  and  lethality  analyses; 

o  reviewed  test  and  evaluation  master  plans,  including  associated 
LFT&E  appendixes  or  attachments,  LFT&E  strategies  and  plans.  Military 
Department  LFT&E  reports,  LFT&E  program  budgets,  and  detailed  program 
schedules  for  selected  programs; 

o  identified  the  roles  of  the  Office  of  the  DOT&E  and  appropriate 
LFT&E  management  groups  within  each  Military  Department  relative  to  the 
DoD  live-fire  test  and  evaluation  process; 

o  determined  and  reviewed  the  Instimte  for  Defense  Analyses' 
involvement  in  the  LFT&E  process; 

0  assessed  the  percentage  of  a  program's  total  research,  development, 
test,  and  evaluation  and  procurement  costs  that  a  program  office  spends  on 
LFT&E;  and 
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0  reviewed  the  funding  for  the  Live-Fire  Test  and  Evaluation  Program 
and  the  Operational  Field  Assessment  Program  in  the  Office  of  the  DOT&E. 

In  accomplishing  the  objective,  we  selected  from  the  universe  of  LFT&E 
oversight  systems  13  major  systems  to  review  during  the  audit:  five  from  the 
Army,  three  from  the  Navy,  and  five  from  the  Air  Force.  We  subjectively 
selected  the  programs  to  provide  a  variety  of  weapon  types  and  a  cross-section 
of  DoD  acquisition  organizations  while  considering  the  amount  of  prior  audit 
coverage.  The  programs  were  in  various  phases  of  the  acquisition  process, 
ranging  from  program  definition  and  risk  reduction  to  production, 
fielding/deployment,  and  operational  support.  The  13  major  systems  selected 
were: 


o  the  Army  Command  and  Control  Vehicle,  the  M2A3  Bradley  Fighting 
Vehicle,  the  MH-47E  and  MH-60K  Special  Operations  Aircraft  (Chinook  and 
Black  Hawk  helicopters),  the  Sense  and  Destroy  Armor  submunition,  and  the 
Wide  Area  Munition  smart  munition; 

o  the  Navy  Air-to-Air  Intercept  Missile-9X  Sidewinder  missile,  the 
Arleigh  Burke  (DDG-51)  Class  Destroyer,  and  the  Joint  Standoff  Weapon 
(Baseline  Version);  and 

o  the  Air  Force  B-2  Spirit  aircraft,  the  C-130J  Hercules  aircraft,  the 
F-16  Close- Air  Support  Aircraft/Battlefield  Air  Interdiction  aircraft,  the  F-22 
Aircraft,  and  the  Joint  Air-to-Surface  Standoff  Missile. 

Appendix  H  describes  the  13  programs. 

To  determine  how  the  Office  of  the  DOT&E  managed  the  preparation  of  the 
LFT&E  report  that  DOT&E  prepares  for  the  Office  of  the  Secretary  of  Defense 
and  Congress,  we  selected  for  review  the  four  reports  that  DOT&E  prepared 
since  the  Federal  Acquisition  Streamlining  Act  of  1994  required  DOT&E  to 
monitor  and  review  live-fire  testing.  Those  reports  were  for  the  M903  Caliber 
0.50  Saboted  Light  Armor  Penetrator  Anununition,  September  1995;  the 
AH-64D  Longbow  Apache  Attack  helicopter,  October  1995;  the  C-17 
Globemaster  III  airlift  aircraft,  November  1995;  and  the  M430A1  40mm 
Cartridge,  May  1996.  For  the  AH-64D  helicopter  and  the  C-17  aircraft, 
DOT&E  combined  the  beyond  low-rate  initial  production  report  with  the 
LFT&E  report  because  those  programs  were  subject  to  DOT&E  operational  and 
live-fire  test  and  evaluation  oversight.  We  examined  those  LFT&E  reports, 
compared  those  reports  to  the  Military  Department  live-fire  test  reports,  and 
reviewed  the  LFT&E  report  preparation  process  at  the  Office  of  the  Secretary  of 
Defense  and  the  Institute  for  Defense  Analyses. 

Auditing  Standards.  We  conducted  this  program  audit  in  accordance  with 
auditing  standards  issued  by  the  Comptroller  General  of  the  United  States,  as 
implemented  by  the  Inspector  General,  DoD.  We  included  such  tests  of 
management  controls  as  we  deemed  necessary.  We  did  not  rely  on  computer- 
processed  data  to  develop  conclusions  on  diis  audit.  Congressional  affairs 
specialists  from  the  Office  of  Congressional  Liaison,  Inspector  General,  DoD, 
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and  technical  experts  from  the  Quantitative  Methods  Division  of  the  Analysis, 
Planning,  and  Technical  Support  Directorate,  Inspector  General,  DoD,  assisted 
in  the  audit. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  DoD;  the  Senate  Subcommittee  on  Defense,  Committee  on 
Appropriations;  the  Senate  Committee  on  Armed  Services;  the  House 
Subcommittee  on  National  Security,  Committee  on  Appropriations;  the  House 
Subcommittee  on  Military  Research  and  Development,  Committee  on  National 
Security;  Advanced  System  Development,  Incorporated;  the  Institute  for 
Defense  Analyses;  International  Development  and  Resources,  Incorporated;  and 
Lockheed  Martin.  Further  details  are  available  upon  request. 


Management  Control  Program 


The  DoD  Directive  5010.38,  "Management  Control  (MC)  Program," 
August  26,  1996,  requires  DoD  managers  to  implement  a  comprehensive  system 
of  management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  Review  of  Management  Control  Program.  We  limited  our  review 
because  of  relevant  coverage  in  Inspector  General,  DoD,  Report  No.  96-028, 
"Implementation  of  the  DoD  Management  Control  Program  for  Major  Defense 
Acquisition  Programs,"  November  28,  1995.  The  report  discusses  the 
effectiveness  of  die  management  control  program  that  the  Defense  Acquisition 
Executive  and  the  Component  Acquisition  Executives  used  for  major  Defense 
acquisition  programs.  The  report  concludes  that  the  acquisition  community  had 
not  effectively  integrated  DoD  Management  Control  Program  requirements  into 
its  management  assessment  and  reporting  processes.  As  a  result  of  the  report 
recommendations,  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  integrated  DoD  Directive  5010.38  requirements  into  the  March  15, 
1996,  revisions  to  DoD  Directive  5000.1,  "Defense  Acquisition,"  and  DoD 
Regulation  5000. 2-R.  Acquisition  managers  are  now  to  use  program  cost, 
schedule,  and  performance  parameters  as  control  objectives  to  implement  the 
DoD  Directive  5010.38  requirements.  The  managers  are  to  identify  material 
weaknesses  through  deviations  from  approved  acquisition  program  baselines  and 
exit  criteria  in  the  "Defense  Acquisition  Executive  Summary"  report. 
Accordingly,  we  limited  our  review  to  management  controls  of  the  LFT&E 
process  at  the  DOT&E  Live  Fire  Test  and  Evaluation  Office  and  the  13  major 
systems  reviewed  in  Appendix  H. 

Adequacy  of  Management  Controls.  We  identified  material  management 
control  weaknesses  for  the  LFT&E  process  at  the  DOT&E  Live  Fire  Test  and 
Evaluation  Office  as  defined  by  DoD  Directive  5010.38.  The  DOT&E  Live 
Fire  Test  and  Evaluation  Office  management  controls  for  the  LFT&E 
administrative  process  did  not  ensure  that  all  programs  subject  to  LFT&E 
requirements  were  correctly  identified  as  LFT&E  candidates  and  placed  on  the 
annual  LFT&E  oversight  list  and  did  not  ensure  that  LFT&E  Office 
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administrative  policy  and  procedures  were  established.  Recommendation  A.,  if 
implemented,  will  help  ensure  that  DOT&E  correctly  and  in  a  timely  manner 
identifies  all  programs  as  LFT&E  candidates.  Recommendation  D.,  if 
implemented,  will  ensure  that  DOT&E  completes  and  issues  the  "Director, 
Operational  Test  and  Evaluation,  Notebook"  for  action  officers  to  include 
written  policies  and  procedures  to  describe  and  document  the  LFT&E 
management  control  process.  We  will  provide  a  copy  of  this  report  to  the 
senior  official  responsible  for  management  controls  in  the  Office  of  the 
DOT&E. 

Adequacy  of  Managemeut's  Self-Evaluatiou.  DOT&E  Live  Fire  Test  and 
Evaluation  Office  officials  identified  LFT&E  as  an  assessable  unit  and,  in  our 
opinion,  correctly  identified  the  risk  associated  with  LFT&E  as  low.  However, 
in  their  evaluation,  the  DOT&E  officials  did  not  identify  the  specific  material 
management  control  weaknesses  that  the  audit  identified  because  the  DOT&E 
evaluation  did  not  specifically  address  the  lack  of  documented  LFT&E  Office 
administrative  policies  and  procedures  or  the  LFT&E  file  system  used  to 
maintain  documentation  to  support  conclusions  and  recommendations  for 
individual  LFT&E  programs  subject  to  LFT&E  program  oversight.  However, 
the  DOT&E  evaluation  did  evaluate  management  attitude;  organizational 
structure;  personnel;  delegation  and  communication  of  auAority  and 
responsibility;  DoD  policies  and  procedures  but  not  specifically  LFT&E  Office 
policies  and  procedures;  planning,  budgeting,  and  accounting;  organizational 
checks  and  balances;  automated  data  processing  considerations;  contracts; 
financial  reporting;  impact  outside  the  agency;  degree  of  centralization;  general 
and  specific  standards;  and  audit  resolution. 


Matter  of  Concern 


During  the  audit,  we  determined  whether  the  Military  Departments  conducted 
verification,  validation,  and  accreditation  (VV&A)  of  modeling  and  simulation 
applications  that  they  used  for  survivability  and  lethality  analyses.  However, 
the  area  did  not  result  in  a  finding  or  recommendation  because  Inspector 
General,  DoD,  Report  No.  93-060,  "Duplication/Proliferation  of  Weapon 
Systems'  Modeling  and  Simulation  Efforts  Within  DoD,"  March  1,  1993, 
addressed  W&A  of  modeling  and  simulation  applications.  Tlie  report  states,  in 
part,  that  the  Military  Departments  had  not  verified,  validated,  or  accredited  the 
majority  of  models  and  simulations  used  in  DoD.  The  lack  of  VV&A  occurred 
because  the  DoD  had  no  requirement,  criterion,  or  standard  by  which  to 
accomplish  a  W&A  process.  In  response,  DoD  issued  DoD  Directive 
5000.59,  "DoD  Modeling  and  Simulation  (M&S)  Management,"  January  4, 
1994,  that  requires  DoD  components  to  establish  W&A  policies,  procedures, 
and  guidelines  for  modeling  and  simulation  applications.  The  directive  states 
that  the  DoD  Component  is  to  accredit  the  modeling  and  simulation  applications 
used  to  support  major  DoD  decisionmaking  organizations  and  processes  for  its 
own  forces  and  capabilities.  Further,  the  directive  requires  the  DoD 
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Component  to  assign  a  DoD  modeling  and  simulation  executive  agent  to 
establish  VV&A  procedures  for  applying  common  or  general  use  models  and 
simulations. 

During  our  live-fire  test  and  evaluation  audit,  we  noted  that  the  Military 
Departments  were  using  modeling  and  simulation  to  supplement  live-fire  tests 
and  evaluations  but  had  not  conducted  W&A  of  the  computer  models  and 
simulations  used  for  live-fire  survivability  and  lethality  analyses  of  weapons 
systems.  However,  for  19  of  the  26  models  associated  with  the  programs 
reviewed  during  the  audit,  the  respective  Military  Departments  were  planning  to 
verify,  validate,  and  accredit  Ae  models  before  using  them  for  live-fire 
survivability  or  lethality  analyses.  For  two  of  the  seven  remaining  models,  the 
Office  of  the  Director,  Test,  Systems  Engineering  and  Evaluation,  and  the 
respective  Military  Department  responsible  for  VV&A  of  the  models  lacked 
funding  to  verify,  validate,  and  accredit  the  models.  For  three  of  the  seven 
models,  the  respective  Military  Departments  stated  that  the  models  had 
imdergone  some  form  of  informal  W&A;  however,  the  Military  Departments 
were  not  planning  to  conduct  additional  W&A.  For  die  remaining 
two  models,  the  responsible  Military  Department  did  not  have  plans  to  use  one 
of  the  models  after  its  current  application  and  did  not  believe  that  accreditation 
was  necessary  for  the  other  model  because  the  Military  Department  used  the 
model  extensively.  Nonetheless,  the  responsible  Military  Department  should 
have  conducted  W&A  of  the  models.  Without  the  Military  Departments 
conducting  the  VV&A,  the  models  may  not  accurately  represent  the  developer's 
conceptual  description  and  specifications  and  may  not  be  an  accurate 
representation  of  real-world  results. 
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During  the  last  5  years,  the  General  Accounting  Office  and  the  Military 
Department  audit  agencies  have  not  issued  reports  specifically  addressing  the 
effectiveness  of  the  DoD  implementation  of  live-fire  testing  requirements  for 
major  Defense  systems  in  Section  2366  of  title  10,  United  States  Code. 
However,  the  Inspector  General,  DoD,  in  1997  and  the  National  Research 
Council  in  1993  and  1995  issued  reports  on  live-fire  testing  issues. 


Inspector  General,  DoD 


Inspector  General,  DoD,  Report  No.  97-064,  "Air-to-Air  Intercept  Missile-9X 
Program,"  January  10,  1997,  discusses  the  adequacy  of  the  program 

management  of  the  Air-to-Air  Intercept  Missile-9X  Program.  The  audit 
concluded  lhat,  although  the  program  was  still  early  in  development,  the 
Program  Office  was  effectively  managing  the  acquisition.  Concerning  live-fire 
testing,  the  report  states  that  the  Program  Manager  required  the  contractor  to 
validate  models  and  simulations  at  least  quarterly  with  data  obtained  from  flight 
and  subsystem  tests.  Accordingly,  the  Program  Office  planned  to  use  actual 
missile  system  and  subsystem  test  and  evaluation  results  to  determine  the 
efficacy  of  modeling  and  simulation  demonstrations,  and  the  results  from 
modeling  and  simulation  demonstrations  will  supplement  actual  test  and 
evaluation  results  used  to  determine  operational  effectiveness  and  suitability. 


National  Research  Council 


National  Research  Council  Report,  "Live  Fire  Testing  of  the  F-22,"  July  1995, 
discusses: 

o  the  practicality  of  full-up,  system-level  survivability  testing  for  the 
F-22  Aircraft  system; 

o  the  implications  regarding  the  affordability  of  the  F-22  System 
Program  Office  conducting  or  not  conducting  the  survivability  tests,  including 
an  assessment  of  the  potential  life-cycle  benefits  that  could  be  derived  from  full- 
up,  system-level  survivability  testing  in  comparison  to  the  cost  of  such  testing; 

o  the  changes  that  have  occurred  to  the  F-22  Aircraft  system  since 
completion  of  the  engineering  and  manufacturing  development  decision  program 
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review  to  cause  the  program  manager  to  request  a  waiver  from  full-up,  system- 
level  survivability  tests  for  the  F-22  Aircraft  system  that  was  not  requested  at 
the  time  of  the  program  review;  and 

0  the  sufficiency  of  the  F-22  Aircraft  system  testing  plans  to  fulfill  the 
requirements  and  purposes  of  Section  2366  of  title  10,  United  States  Code. 

The  report  concludes  that  the  F-22  System  Program  Office  could  conduct  live- 
fire  testing  at  levels  below  full-up,  system-level  survivability  testing,  so  a 
request  for  waiver  from  full-up,  system-level  survivability  testing  for  the  F-22 
Aircraft  system  was  justified.  The  report  recommended  additional  live-fire  tests 
for  components  and  continued  DoD  efforts  to  develop  viable  cost-benefit 
methodologies  for  planning  the  extent  of  live-fire  testing. 

National  Research  Council  Report,  "Vulnerability  Assessment  of  Aircraft,  A 
Review  of  tiie  Department  of  Defense  Live  Fire  Test  and  Evaluation  Program," 
January  1993,  reviewed  current  methodologies  that  the  Army,  the  Navy,  and 
the  Air  Force  used  to  determine  the  vulnerability  of  airborne  systems  to  enemy 
conventional  weapons;  evaluated  the  cost  and  effectiveness  of  those 
methodologies;  evaluated  the  most  appropriate  methodologies  for  the 
application;  and  reviewed  the  current  direction  on  live-fire  testing  and 
evaluation.  The  report  concludes  that: 

0  the  live-fire  test  law  (Section  2366  of  title  10,  United  States  Code) 
was  satisfactory  in  its  present  form  because  of  the  waiver  process; 

o  the  1987  live-fire  test  law  mandates  live-fire  testing  of  full-scale,  full- 
up  aircraft,  including  on-board  ordnance,  unless  the  Office  of  the  Secretary  of 
Defense  approved  a  request  for  waiver; 

0  the  1988  LFT&E  guidelines  and  the  1989  LFT&E  planning  guide 
were  not  consistent  with  the  live-fire  test  law;  and 

o  the  implied  intent  of  the  live-fire  test  law  was  to  force  the 
consideration  of  vulnerability  during  the  design  process. 
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VNrrKD  STATES  CODE  ANNOTATED 
TITJLEIO,  ARMED  FORCES 
StJm  iTLE  A~GENERAL  MILITARY  LAW 
HART  IV-SERVICE  SUPPLY,  AND  PROCUREMENT 
CHAPTER  r39-RES£ARCH  AND  DEVELOPMEN'I* 


Current  through  P.L.  105-15,  approved  5-15-97 

§  2366.  Major  systems  and  munitions  programs:  survivabilUy  testing  and  lethality  testing  required  before  full- 
scale  production 

(a)  Requii'cmcuts,— (1)  The  Secretary  of  Defense  shall  provide  that— 

(A)  a  covered  system  may  not  proceed  beyond  low-rate  initial  production  until  realistic  survivability  testing  of 
the  system  is  completed  in  accordance  with  this  section  and  the  report  required  by  subsection  (<I>  with  respect  to 
that  testing  is  submitted  in  accordance  with  that  subsection;  and 

(B)  a  major  munition  program  or  a  missile  program  may  not  proceed  beyond  low-raie  initial  production  until 
realistic  lethality  testing  of  the  program  is  completed  in  accordance  with  this  section  ajid  tlte  report  required  by 
subsection  (d)  with  respect  to  that  testing  is  submitted  in  accordance  with  that  subsection. 

(2)  The  Secretary  of  Defense  shall  provide  that  a  covered  product  improvemetu  prograTU  may  not  proceed 
beyond  iow>rate  initial  production  until— 

(A)  in  the  case  of  a  product  improvemenr  to  a  covered  system,  realistic  surviva)>iTiiy  testing  is  ciimplctcd  in 
accordance  with  this  section;  and 

(B)  in  the  case  of  a  product  improvement  to  a  major  munitions  program  or  a  missile  program,  realistic 
lethality  testing  is  completed  in  accordance  with  this  section. 

Test  guidelines,— (1)  Survivability  :ind  lethality  tests  required  under  subsection  (a)  shall  be  carried  out 
sufficiently  early  in  the  development  phase  of  the  system  or  program  (including  a  covered  product  improvement 
program)  to  allow  any  design  dcficitmcy  demonstrated  by  the  testing  to  be  corrected  in  the  design  of  the  system, 
rnunilion.  or  missile  (or  in  the  product  modification  or  upgrade  to  the  system,  munition,  or  missile)  before 
proceeding  beyond  low^raic  initial  production. 

.(2)  The  costs  of  ail  tests  required  under  that  subsection  shall  be  paid  from  fui^ds  available  for  the  system  being 
tested. 

(c)  Waiver  authority.— (1)  The  Secretary  of  Defense  may  waive  the  application  of  the  survivability  and  lethality 
tests  of  this  section  to  a  covered  system,  munitions  program,  missile  program,  or  covered  product  improvement 
program  if  the  Secretary,  before  the  system  or  program  entem  engineering  ancl  manufacturing  dcvel0t)tncnt, 
certifies  to  Congress  that  live-fire  testing  of  such  system  or  pn>gram  would  be  unreasonably  exf>en.<;ive  and 
Impractical. 

(2)  In  ihc  case  of  a  covered  system  (or  covered  product  itnprovement  program  for  a  covered  system),  the 
Secretary  may  waive  (he  application  of  the  survivability  and  lotbaUty  lesis  of  this  section  to  such  system  or 
program  and  instead  allow  testing  of  the  system  or  program  in  combat  by  filing  munitions  likely  to  be 
encountered  in  combat  at  components,  subsystems,  and  suba.sscmb1ics,  mgcihcr  with  performing  design  analyses, 
modeling  and  simulation,  and  analysis  of  combat  data.  Such  alternative  testing  may  not  be  carried  out  in  the  case 
of  any  covered  system  (or  covered  product  improvement  program  for  a  covered  system)  unless  the  Secretary 
certifies  to  Congress,  before  the  ^sicm  or  program  enters  engineering  and  manufacturing  devetopmetu,  that  the 
survivability  and  ieihality  testing  of  such  system  or  program  otherwise  required  by  this  section  would  be 
unrea.<u)nably  expensive  and  impracticable. 
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(3)  The  Secretary  shall  include  with  any  certification  under  paragraph  CO  or  ('2)  a  report  explaining  how  the 
Secretary  plans  to  evaluate  the  survivability  or  the  lethality  of  the  syatctn  or  pmgrani  and  assessing  possible 
aUematives  to  realistic  survivability  testing  of  the  system  or  program. 

(4)  In  time  of  war  or  mobilization,  (he  President  may  suspend  the  operation  of  any  provision  uf  this  seelion. 

(d)  Reporting  to  Congress^At  the  conclusion  of  survivabilily  or  lethality  testing  under  subsection  (a),  die 
Secretary  of  Defense  shall  submit  a  report  on  the  testing  to  (he  the  congressional  defense  committees.  JBacb  such 
Tcpon  shall  describe  the  results  of  the  survivability  or  lethality  testing  and  shall  give  the  Secretary's  overall 
assessment  of  the  testing. 

(c)  Definitions.— In  this  Section: 

Cl)  The  term  *covcred  system*  means  a  vehicle,  weapon  platform,  or  convcnliunal  weapon  system— 

(A)  that  includes  features  designed  to  provide  some  degree  of  protection  to  users  in  combat;  and 
<B>  that  is  a  major  system  wiUiin  the  meaning  of  that  term  in  sex^tion  2302(5)  of  this  title. 

<2)  The  term  -major  munitions  program”  means— 

(A>  a  munition  program  for  which  more  than  1 ,000,000  rounds  are  planned  to  be  acquired;  or 

(B)  a  conventional  munitions  program  that  is  a  major  system,  within  the  meaning  of  that  term  in  section 
2302(5)  ofUlis  title. 

(3)  T'he  term  “realistic  survivability  testing"  means,  in  the  case  of  a  covered  system  (or  a  covered  product 
improvement  program  for  a  covered  system),  testing  for  vulnerability  of  the  system  in  combat  by  firing 
munitions  likely  to  be  encountered  in  combat  (or  munitions  with  a  capability  similar  to  such  munitions)  at  the 
$3rstem  configured  for  combat,  with  the  primary  emphasis  on  testing  vulnerability  with  respect  to  potential  user 
casualties  and  taking  into  equal  consideration  the  susceptibility  to  altack  and  combat  pcrformancjc  of  the  system. 

(4)  The  term  "realistic  Icihaliiy  testing"  means,  in  the  case  of  a  major  munitions  program  or  a  missile  program 
(or  a  covered  product  Improvement  program  for  such  a  program),  testing  for  letliality  by  fii'lng  the  munition  or 
missile  concerned  at  appropriate  targets  configured  for  combat. 

(5)  The  term  "cundguicd  for  combat ",  with  respect  to  a  weapon  system,  plaifurm,  or  vehicle,  means  loaded 
or  equipped  witli  all  dangerous  materials  (including  all  nammables  and  explosives)  dial  would  normally  be  on 
hoard  in  enmhat. 

(6)  The  term  "covered  product  improvement  program"  means  a  program  under  which— 

(A)  a  modification  or  upgrade  will  be  made  to  a  covered  sy-%tem  which  (as  determined  by  the  Secretary  of 
Defense)  is  likely  to  affect  significantly  the  survivability  of  such  system;  or 

(B)  a  modiilcation  or  upgrade  will  be  nude  to  a  niigor  munitions  program  or  a  missile  program  which  (as 
determined  by  the  Secretary  of  Defense)  Is  likely  to  affect  significantly  the  lethality  of  (he  munition  or  missile 
produced  under  the  program. 

(7)  The  term  "congressiona]  defense  commilLees"  mcans><. 

<A)  the  Committee  on  Armed  Services  and  the  Coniinittee  on  Appropriations  of  die  Senate;  and 

(B)  the  Commit  lee  on  National  Security  and  die  Coinmiuee  <in  Appropriations  of  the  House  of 
Representatives. 


^Subsection  2302(5)  of  title  10,  United  States  Code,  "Definitions,”  provides  that  "The  term  'major 
system’  means  a  combination  of  elements  that  will  function  together  to  produce  the  capabilities  required 
to  fulfill  a  mission  need.  The  elements  may  include  hardware,  equipment,  software  or  any  combination 
thereof,  but  excludes  construction  or  other  improvements  to  real  property.  A  system  shall  be  considered 
a  major  system  if  (A)  the  conditions  of  section  2302d  of  this  title  are  satisfied,  or  (B)  the  system  is 
designated  a  ’major  system'  by  the  head  of  the  agency  responsible  for  the  system. " 
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Accreditation.  Accreditation  is  the  official  certification  that  a  model  or 
simulation  is  acceptable  to  use  for  a  specific  purpose. 

Acquisition  Category.  An  acquisition  category  is  an  attribute  of  an  acquisition 
program  that  determines  the  program's  level  of  review,  decision  authority,  and 
applicable  procedures.  The  acquisition  categories  consist  of  I,  major  Defense 
acquisition  programs;  I  A,  major  automated  information  systems;  II,  major 
systems;  and  III,  all  other  acquisition  programs. 

Commercial  Item.  A  commercial  item  is  any  item,  other  than  real  property, 
that  is  of  a  type  customarily  used  for  nongovernmental  purposes  and  that  has 
been  sold,  leased,  or  licensed  to  the  general  public  or  has  been  offered  for  sale, 
lease,  or  license  to  the  general  public.  A  commercial  item  is  also  any  item  that 
evolved  through  advances  in  technology  or  performance  and  that  is  not  yet 
available  in  the  commercial  marketplace,  but  will  be  available  in  the  commercial 
marketplace  in  time  to  satisfy  the  delivery  requirements  under  a  Government 
solicitation. 

Configured  for  Combat.  Configured  for  combat  refers  to  a  weapon  system, 
platform,  or  vehicle  loaded  or  equipped  with  all  dangerous  materials,  including 
all  flammables  and  explosives,  that  would  normally  be  on  board  in  combat. 

Covered  Product  Improvement  Program.  A  covered  product  improvement 
program  is  a  modification  or  upgrade  to  a  covered  system  or  to  a  major 
munitions  program  or  a  missile  program  that  the  Secretary  of  Defense  has 
determined  will  likely  affect  significantly  the  survivability  or  lethality, 
respectively,  of  the  system  or  of  Ae  munition  or  missile  produced  under  the 
program. 

Covered  System.  A  covered  system  is  any  vehicle,  weapon  platform,  or 
conventional  weapon  system  that  includes  features  designed  to  provide  some 
degree  of  protection  to  Users  in  combat  and  that  is  a  major  system. 

Detailed  Live-Fire  Test  and  Evaluation  Plan.  A  detailed  live-fire  test  and 
evaluation  plan  is  a  document  or  series  of  documents  that  implement  the  live- 
fire  test  and  evaluation  strategy  and  that  describe  the  detailed  test  procedures, 
test  conditions,  data  collection,  and  analysis  process  used  during  &e  live-fire 
test. 

Detailed  Live-Fire  Test  and  Evaluation  Report.  A  detailed  live-fire  test  and 
evaluation  report  provides  the  results  and  overall  evaluation  of  all  testing 
identified  in  the  live-fire  test  and  evaluation  strategy. 

Developing  Organization.  A  developing  organization  is  the  command 
responsible  for  research,  development,  and  production  of  a  new  item. 
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Full-Materiel  Release  Process.  The  full-material  release  process  assures  that 
Army  materiel  is  suitable  and  supportable  before  release  to  users.  The  program 
executive  officer  and  the  project  manager  authorize  full-materiel  release  when 
the  materiel  meets  all  established  requirements  and  specifications  including  the 
final  materiel  fielding  plan,  the  materiel  fielding  agreement,  the  net  equipment 
training  provisions,  the  logistics  support  system,  the  authenticated  equipment 
publications,  the  basic  sustainment  materiel  needed  for  initial  support,  and  the 
elimination  or  acceptance  of  identified  system  hazards. 

Full-Up  Live-Fire  Test.  A  full-up  live-fire  test  is  a  vulnerability  test  of  a 
complete  or  partial  system  loaded  or  equipped  with  all  dangerous  materials, 
including  flammables  and  explosives,  that  would  normally  be  on  board  in 
combat.  All  critical  subsystems  that  could  contribute  to  the  test  outcome  must 
be  operating  under  realistic  conditions.  For  lethality  testing,  the  munitions  or 
missile  must  be  production-representative.  The  target  must  be  representative  of 
the  class  of  systems  that  includes  the  threat  and  must  be  sufficiently  realistic  to 
demonstrate  the  lethal  effects  that  the  weapon  is  designed  to  produce. 

Full-Up,  System-Level,  Live-Fire  Test.  A  full-up,  system-level,  live-fire  test 
is  live-fire  testing  that  fully  satisfies  the  statutory  requirement  for  realistic 
survivability  or  realistic  lethality  testing. 

Integrated  Product  Team.  An  integrated  product  team  is  a  group  of  selected 
individuals  representing  multiple  disciplines  formed  to  produce  a  specific 
product  or  service.  The  individuals  selected  have  mumal  as  well  as  individual 
accountability;  contribute  to  integrated,  concurrent  decisionmaking;  and  are 
empowered  within  defined  limits  to  decide  and  act  to  ensure  the  realization  of 
the  specific  product  or  service. 

Lethality.  Lethality  is  the  ability  of  a  munition,  laser,  or  high  power 
microwave  to  cause  damage  that  will  result  in  the  loss  or  degradation  in  the 
ability  of  a  target  system  to  complete  its  mission. 

Live-Fire  Test.  A  live-fire  test  is  a  test  conducted  in  accordance  with  an  Office 
of  the  Secretary  of  Defense-approved  live-fire  test  and  evaluation  strategy  that 
involves  the  firing  of  acmal  munitions  at  target  components,  target  subsystems, 
target  subassemblies  or  system-level  targets,  which  may  or  may  not  be 
configured  for  combat,  to  examine  personnel  casualty,  vulnerability,  or  lethality 
issues,  or  all  three. 

Live-Fire  Test  and  Evaluation.  Live-fire  test  and  evaluation  is  full-up, 
system-level  survivability  or  lethality  testing  of  a  covered  system,  major 
munition  program,  missile  program,  or  product  improvement  to  a  covered 
system,  major  munition  program,  or  missile  program  before  it  can  proceed 
beyond  low-rate  initial  production. 

Live-Fire  Test  and  Evaluation  Strategy.  A  live-fire  test  and  evaluation 
strategy  is  a  part  of  the  test  and  evaluation  master  plan  describing  who,  what, 
why,  when,  where,  and  how  the  program  manager  plans  to  satisfy  the  live-fire 
test  and  evaluation  requirement. 
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Low-Rate  Initial  Production.  Low-rate  initial  production  is  the  production  of 
a  system  in  limited  quantity  to  provide  articles  for  operational  test  and 
evaluation  and  to  establish  an  initial  production  rate  sufficient  to  lead  to  fiill-rate 
production  upon  successful  completion  of  operational  testing. 

Major  Munitions  Program.  A  major  munitions  program  is  a  munitions 
program  for  which  DoD  plans  to  acquire  more  than  one  million  rounds  or  a 
conventional  munitions  program  that  is  a  major  system. 

Major  System.  A  major  system  is  a  system  that  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  has  estimated  will  require  an  eventual 
total  expenditure  for  research,  development,  test,  and  evaluation  of  more  than 
$75  million  in  FY  1980  constant  dollars  (approximately  $140  million  in 
FY  1996  constant  dollars)  or  for  procurement  of  more  than  $300  million  in 
FY  1980  constant  dollars  (approximately  $645  million  in  FY  1996  constant 
dollars).  A  major  system  is  synonymous  with  an  Acquisition  Category  II 
program. 

Model.  A  model  is  a  physical,  mathematical,  or  otherwise  logical 
representation  of  a  system,  entity,  phenomenon,  or  process. 

Nondevelopmental  Item.  A  nondevelopmental  item  is  an  item  that  meets  one 
of  the  following  criteria: 

o  any  previously  developed  item  of  supply  used  exclusively  for 
governmental  purposes  by  a  Federal  agency,  a  State  or  local  government,  or  a 
foreign  government  wi&  which  the  United  States  has  a  mutual  defense 
cooperation  agreement; 

o  any  item  satisfying  the  previous  criteria  but  that  requires  only  minor 
modification  or  modifications  of  the  type  customarily  available  in  the 
conunercial  marketplace  to  meet  the  requirements  of  the  procuring  department 
or  agency;  or 

o  any  item  produced  for  Governmental  purposes  described  above  but 
not  yet  in  use. 

Realistic  Lethality  Testing.  Realistic  lethality  testing  is  testing  to  determine 
the  lethality  of  a  major  munitions  program,  missile  program,  or  a  covered 
product  improvement  program  by  firing  the  munition  or  missile  concerned  at 
appropriate  targets  configured  for  combat. 

Realistic  Survivability  Testing.  Realistic  survivability  testing  is  testing  for 
vulnerability  of  a  covered  system,  or  product  improvement  to  a  covered  system, 
in  combat  by  firing  munitions  likely  to  be  encountered  in  combat  (or  munitions 
with  similar  capability)  at  the  system  configured  for  combat,  with  the  primary 
emphasis  on  testing  vulnerability  with  respect  to  potential  user  casualties  and 
taking  into  equal  consideration  the  susceptibility  to  attack  and  combat 
performance  of  the  system. 
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Request  for  Waiver.  A  request  for  waiver  is  a  request  to  exempt  a  system 
from  full-up,  system-level  live-fire  testing.  The  waiver  request  includes  a 
strategy  that  explains  how  the  Military  Department  plans  to  evaluate  the 
vulnerability  or  lethality  of  the  system  and  assesses  possible  alternatives  to  full- 
up,  system-level  live-fire  testing. 

Simulation.  Simulation  is  a  method  for  implementing  a  model  over  time. 
Simulation  is  also  a  technique  for  testing,  analysis,  or  training  in  which  testers 
use  real-world  systems  or  in  which  a  model  reproduces  real-world  and 
conceptual  systems. 

Survivability.  Survivability  is  the  capability  of  a  system  and  crew  to  avoid  or 
withstand  a  man-made  hostile  environment  without  suffering  an  abortive 
impairment  to  their  ability  to  accomplish  their  mission. 

Test  and  Evaluation  Master  Plan.  A  test  and  evaluation  master  plan  is  a 
document  that  identifies  and  integrates  objectives,  responsibilities,  resources, 
and  schedules  for  developmental,  operational,  and  live-fire  test  and  evaluation 
actions  that  a  program  is  to  accomplish  before  milestone  decision  points. 

Validation.  Validation  is  a  process  to  determine  the  degree  to  which  a  model  is 
an  accurate  representation  of  the  real-world  from  the  perspective  of  the  intended 
users  of  the  model. 

Verification.  Verification  is  a  process  to  determine  whether  a  model 
implementation  accurately  represents  the  developer's  conceptual  description  and 
specifications. 

Vulnerability.  Vulnerability  is  the  characteristic  of  a  system  to  sustain  a  loss  or 
reduction  of  capability  to  perform  its  mission  as  a  result  of  the  effects  of  a  man¬ 
made  hostile  environment.  Vulnerability  is  a  subset  of  survivability. 
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Our  detailed  response  to  the  specific  comments  from  the  Principal  Deputy, 
Assistant  Secretary  of  the  Air  Force  (Acquisition),  and  the  Director,  Air  Force 
Test  and  Evaluation  Directorate,  on  statements  in  the  draft  report  follows.  The 
complete  texts  of  those  comments  are  in  Part  III. 

Principal  Deputy,  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
Comments.  The  Principal  Deputy  stated  that  the  draft  report’s  description  of 
the  B-2  Spirit  bomber  should  be  corrected  for  accuracy.  He  provided  a 
description  of  the  B-2  Spirit  bomber. 

Audit  Response.  The  report  now  shows  the  Principal  Deputy's  description  of 
the  B-2  Spirit  bomber. 

Director,  Air  Force  Test  and  Evaluation  Directorate,  Comments.  The 
Director,  Air  Force  Test  and  Evaluation  Directorate,  (the  Director)  provided 
specific  comments  on  the  term  major  system,  the  Internet  website,  die  DoD 
Regulation  5000. 2-R  revision,  requests  for  C-17  LFT«&:E  technical  report, 
placement  of  LFT&E  under  DOT&E,  DOT&E  responsibilities,  and  the  issue  of 
including  the  live-fire  law  in  the  report.  The  follow  discuss  those  specific 
comments  and  the  audit  response. 

Term  Major  System.  The  Director  stated  that  the  report  uses  the  term 
major  Defense  system  instead  of  major  system  to  describe  the  types  of  systems 
that  Section  2366  of  title  10,  United  States  Code,  covers.  He  further  stated  that 
Section  2366  addresses  only  major  systems. 

Audit  Response.  Subsection  2366(e)(1)(B)  of  title  10,  United  States 
Code,  defines  major  system  within  the  meaning  of  that  term  in 
Subsection  2302(5)  of  title  10,  United  States  Code,  "Major  system:  definitional 
threshold  amounts."  Subsection 2302(5)  differentiates  between  DoD  and 
civilian  agenCy  major  systems.  Because  we  have  such  a  wide  variety  of  readers 
of  our  reports,  we  used  the  term  major  Defense  system  in  our  audit  objective 
and  other  appropriate  places  in  the  report  to  let  the  reader  know  that  we  were 
only  addressing  DoD  and  not  civilian  agency  major  systems  in  our  report. 

Internet  Website.  The  Director  suggested  that  we  recommend  that 
DOT&E  put  their  LFT&E  oversight  list  on  die  DOT&E  Internet  website  for 
review  and  comment  before  officially  publishing  the  list. 

Audit  Response.  We  provided  the  Office  of  the  DOT&E  with  a  copy  of 
the  Director's  conunents  for  consideration. 

DoD  Regulation  5000.2-R  Revision.  The  Director  suggested  the 
deletion  of  the  statement  in  the  report  that  the  Office  of  the  Secretary  of  Defense 
was  in  the  process  of  revising  DoD  Regulation  5000.2-R  to  require  commercial 
items,  including  commercial  off-the-shelf  items,  nondevelopmental  items,  and 
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advanced  concept  technology,  demonstrators  or  prototypes  that  meet  the 
definition  of  a  covered  system  or  major  munitions  program  to  be  subject  to 
LFT&E.  The  Director  stated  that  the  change  has  been  withdrawn. 

Audit  Response.  As  of  August  1997,  the  Defense  Acquisition  Policy 
Steering  Group  that  oversees  the  Defense  Acquisition  Policy  Working  Group, 
the  group  responsible  for  changes  to  DoD  Regulation  5000. 2-R,  was  still 
considering  the  revision  for  possible  inclusion  in  future  changes  to  DoD 
Regulation  5000.2-R.  We  revised  the  statement  in  the  report  accordingly. 

Requests  for  C-17  Life-Fire  Test  and  Evaluation  Technical 
Report.  The  Director  stated  that,  despite  several  requests,  the  Air  Force  has 
not  received  the  IDA  technical  report  on  the  C-17  LFT&E.  He  stated  that  the 
technical  report  may  contain  valuable  lessons-leamed  that  could  help  improve 
other  Air  Force  programs. 

Audit  Response.  We  notified  the  Office  of  the  DOT&E  concerning  the 
Director's  request.  The  Office  will  take  appropriate  action. 

Placement  of  Live-Fire  Test  and  Evaluation  Under  the  Director, 
Operational  Test  and  Evaluation.  The  Director  stated  that  the  report 
mistakenly  states  that  the  Federal  Acquisition  Streamlining  Act  (the  Act)  placed 
LFT&E  under  DOT&E.  Further,  the  Director  stated  that  the  Act  only  added  a 
provision  to  Section  139  of  title  10,  United  States  Code,  requiring  DOT&E  to 
"monitor  and  review  live  fire  testing  activities  of  the  Department  of  Defense 
provided  for  under  section 2366  of  this  title."  The  Director  also  stated  that 
DoD  moved  the  LFT&E  office  and  personnel  from  the  Office  of  the  Director, 
Test,  Systems  Engineering  and  Evaluation,  to  the  Office  of  the  DOT&E. 

Audit  Response.  We  did  not  mean  to  imply  that  the  Federal 
Acquisition  Streamlining  Act  moved  the  LFT&E  office  and  personnel  from  the 
Office  of  the  Director,  Test,  Systems  Engineering  and  Evaluation,  to  the  Office 
of  the  DOT&E.  We  modified  the  report  accordingly. 

Director,  Operational  Test  and  Evaluation,  Responsibilities.  The 

Director  stated  that  &e  draft  report  incorrectly  states  that  DOT&E  accomplishes 
its  oversight  responsibilities  by  reviewing  and  approving  live-fire  documents. 
He  stated  that  Subsection  139(b)(6)  of  title  10,  United  States  Code,  only  gives 
DOT&E  authority  to  "monitor  and  review  live  fire  testing  activities"  of  DoD. 
He  stated  that  the  DOT&E  authority  extends  only  to  approving  the  LFT&E 
strategy  and  not  the  LFT&E  plans.  Finally,  he  stated  that  the  last  sentence  of 
the  referenced  paragraph  should  be  revised  to  indicate  that  DOT&E  oversees 
LFT&E  of  a  program. 

Audit  Response.  Concerning  the  DOT&E  authority  to  approve  the 
LFT&E  plans,  DoD  Regulation  5000.2-R  requires  DOT&E  to  approve  Military 
Department  detailed  LFT&E  plans  identified  for  DOT&E  approval  per  the 
matrix  of  LFT&E  phases  included  in  the  test  and  evaluation  master  plan. 
Concerning  the  sentence  about  DOT&E  LFT&E  oversight  of  a  program,  we 
revised  the  sentence  accordingly. 
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Live-Fire  Law.  The  Director  suggested  that  the  report  include  in  an 
appendix  the  actual  live-fire  law,  Section  2366  of  title  10,  United  States  Code, 
because  of  the  complexity  of  the  issues. 

Audit  Response.  We  included  the  text  of  Section  2366  in  Appendix  C 
of  this  report. 
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Live-Fire  Test  and  Evaluation  Process.  After  DOT&E  designates  a  program 
as  an  LFT&E  program,  the  respective  Military  Department  program  manager 
begins  to  plan,  ftod,  and  execute  an  LFT&E  strategy  for  fee  designated 
program.  If  full-up,  system-level  survivability  or  lethality  testing  is 
unreasonably  expensive  and  impractical,  fee  program  manager  is  to  prepare  a 
request  for  waiver  from  full-up,  system-level  survivability  or  lethality  testing. 
The  request  for  waiver  is  to  include  alternative  plans  for  evaluating  fee 
survivability  or  lethality  of  fee  system.  In  fee  test  and  evaluation  master  plan, 
fee  program  manager  is  to  include  a  description  of  fee  LFT&E  program  and  fee 
LFT&E  strategy  for  full-up,  system-level  or  alternative  survivability  or  lethality 
testing.  The  program  manager  also  prepares  a  matrix  document  feat  covers  all 
tests  within  fee  LFT&E  strategy,  fee  test  schedule,  fee  issues  that  fee  tests  will 
address,  and  fee  planning  documents  feat  fee  Military  Department  proposes  for 
submission  to  DOT&E  for  approval  or  information  and  review  purposes.  The 
live-fire  test  reporting  process  diagram  in  this  appendix  illustrates  fee  live-fire 
test  process  and  fee  DOT&E  oversight  and  reporting  processes.  The  objectives 
of  feose  DOT&E  processes,  in  addition  to  meeting  congressional  reporting 
requirements,  are  to: 

o  enable  fee  Secretary  of  Defense  to  make  informed  system  acquisition 
decisions, 

o  enable  fee  program  developers  to  gain  insight  into  potential  design 
flaws  feat  they  can  correct  before  fee  program  enters  fiill-rate  production,  and 

o  ensure  feat  knowledge  of  user  casualties,  system  survivability,  system 
lethality,  or  any  combination  of  them  are  based  on  fiill-up,  system-level  testing 
or  acceptable  alternatives  when  justified  by  a  request  for  waiver  from  full-up, 
system-level  survivability  or  lethality  testing. 

As  fee  LFT&E  program  progresses  through  the  acquisition  process,  fee  Military 
Department  prepares  detailed  test  and  evaluation  plans  for  each  test  as  well  as 
pre-shot  predictions.  Those  plans  describe  fee  detailed  test  procedures,  test 
conditions,  and  data  collection  and  analysis  processes  that  fee  Military 
Department  testers  will  use  when  conducting  each  test.  Upon  completion  of 
each  test,  fee  Military  Department  testers  prepare  a  detailed  test  and  evaluation 
report.  The  Military  Departments  may  prepare  additional  reports  to  record  fee 
results  of  modeling  and  simulations  and  other  analyses.  Those  Military 
Department  live-fire  reports  are  fee  primary  source  of  data  for  both  fee  Military 
Department  and  DOT&E  live-fire  test  evaluations  and  assessments. 

Within  120  days  after  completing  live-fire  testing,  fee  Military  Department 
prepares  an  evaluation  report  feat  contains  fee  results  and  overall  evaluation  of 
all  testing  identified  in  fee  LFT&E  strategy  for  LFT&E  programs,  and  fee 
Military  Department  provides  fee  report  to  DOT&E  as  required  by  DoD 
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Regulation  5000.2-R.  The  Military  Department  may  also  provide  the  report  to 
the  Component  Acquisition  Executive  and  the  Defense  Acquisition  Board,  as 
appropriate. 

OfHce  of  the  Director,  Operational  Test  and  Evaluation,  Oversight 
Process.  The  Office  of  the  DOT&E  accomplishes  its  oversight  responsibilities 
by  reviewing  and  approving  live-fire  documents,  by  observing  live-fire  tests, 
and  by  preparing  an  independent  assessment  report  on  the  system's  survivability 
or  lethality  and  the  adequacy  of  the  live-fire  tests  and  evaluations.  As  part  of 
that  oversight,  DOT&E  reviews  and  approves  Military  Department  test  and 
evaluation  master  plans,  including  LFT&E  strategies  and  plans;  attends  live-fire 
testing;  and  accesses  test  data,  test  reports,  and  other  data  and  reports  from 
models  and  simulations  and  other  analyses.  At  the  completion  of  the  Military 
Department  LFT&E,  DOT&E  prepares  its  independent  assessment  report  using 
analytical  and  technical  support  from  IDA.  If  DOT&E  oversees  the  operational 
and  live-fire  testing  on  the  program,  DOT&E  prepares  a  combined  beyond  low- 
rate  initial  production  and  live-fire  report  on  the  program. 

To  prepare  the  combined  report,  the  DOT&E  operational-test  action  officer, 
who  is  responsible  for  preparing  the  report,  obtains  draft  input  of  the  live-fire 
test  results  and  evaluation  from  IDA.  Ihe  DOT&E  operational-test  action 
officer  briefs  the  results  to  DOT&E  and  his  deputies,  as  appropriate.  After  the 
draft  input  to  DOT&E,  IDA  usually  prepares  a  technical  report  for  DOT&E  that 
formally  reports  its  evaluation  and  the  live-fire  test  results. 

For  programs  subject  only  to  DOT&E  live-fire  oversight,  DOT&E  live-fire-test 
action  officers  obtain  a  draft  technical  report  from  IDA  and  modify  the  report  to 
meet  the  congressional  reporting  requirements  by  adding  an  introduction  and  a 
conclusion  that  includes  an  overall  assessment  of  the  live-fire  results.  Later, 
IDA  usually  prepares  a  final  technical  report  for  DOT&E. 

Using  the  respective  combined  beyond  low-rate  initial  production  and  live-fire 
report  or  die  modified  live-fire  technical  report,  DOT&E  provides  summary 
information  to  the  appropriate  milestone  decision  authorities  before  the 
program’s  full-rate  production  decision.  After  completing  the  combined  beyond 
low-rate  initial  production  and  live-fire  report  or  modified  live-fire  technical 
report,  DOT&E  provides  the  respective  report  to  the  Secretary  of  Defense  and 
Congress  before  Ae  program's  full-rate  production  decision. 
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Live-Fire  Test  and  Evaluation  Reporting  Process  Diagram 
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^Program  managers,  contractoi’s,  and  miiitaiy  department  tacticians  use  test,  simulation, 
and  study  results  to  improve  the  system  design* 

^For  programs  without  operational  lest  oversight,  the  Institute  for  Defense  Analysis 
Technical  Report  would  continue  to  serve  us  the  basis  tor  the  DOT&E  report  to  Congress. 
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The  DOT&E  used  the  IDA  to  make  an  evaluation  of  the  validity  of  the  Military 
Department  live-fire  testing,  analyses,  and  program  results.  After  DOT&E 
became  responsible  for  monitoring  and  reviewing  live-fire  actions,  DOT&E 
issued  two  beyond  low-rate  initial  production  reports  that  combined  operational 
test  results  with  live-fire  test  results  and  assessments.  Those  reports  were  the 
"AH-64D  Longbow  Apache  Attack  Helicopter,"  October  1995,  and  the  "C-17 
Globemaster  III  Airlift  Aircraft,"  November  1995.  Also,  after  becoming 
responsible  for  live-fire  test  and  evaluation  (LFT&E),  DOT&E  live-fire-test 
action  officers  issued  two  reports  for  programs  subject  only  to  DOT&E  live-fire 
oversight.  Those  reports  were  the  "Live  Fire  Test  and  Evaluation  of  the  0.50 
Caliber  M903  Saboted  Light  Armor  Penetrator  (SLAP)  Ammunition," 
September  26,  1995,  and  the  "Live  Fire  Test  and  Evaluation  Report  on  the 
M430A1  40MM  Cartridge,"  May  30,  1996. 

AH-64D  Helicopter  Reporting.  To  properly  report  on  the  Hellfire  Modular 
Missile  System  (Hellfire  Missile)  and  tihie  Modernized  Apache  helicopter  system 
(Longbow  Apache),  DOT&E  merged  those  systems  into  a  single  DOT&E 
combined  beyond  low-rate  initial  production  and  live-fire  report.  The  Army 
Materiel  Systems  Analysis  Activity  issued  two  draft  evaluation  reports, 
"Independent  Evaluation  Report  for  the  Longbow  Hellfire  Modular  Missile 
System  (LBHMMS)  Live  Fire  Test  and  Evaluation,"  July  1995,  and 
"Modernized  Apache  Live  Fire  Vulnerability  Test  Independent  Evaluation 
Report,"  September  1995,  and  provided  those  draft  reports  to  DOT&E.  IDA 
provided  analyses,  charts,  graphs,  and  narrative  on  the  Hellfire  Missile  and 
Longbow  Apache  live-fire  tests  to  the  DOT&E  operational-test  action  officer  to 
support  the  DOT&E  preparation  of  the  live-fire  test  section  of  the  combined 
beyond  low-rate  initial  production  and  live-fire  report.  IDA  also  provided 
DOT&E  with  a  draft  technical  report,  IDA  Paper  P-3092,  "Live  Fire  Test  and 
Evaluation  of  the  AH-64D  Longbow  Apache,"  September  22,  1995.  The  IDA 
draft  technical  report  provided  DOT&E  with  a  basis  for  writing  the  Longbow 
Apache  LFT&E  portions  of  its  combined  beyond  low-rate  initial  production  and 
live-fire  report.  However,  IDA  had  not  conducted  a  peer  review  of  its  draft 
technical  report  before  it  released  the  report  to  DOT&E.  Further,  IDA  did  not 
issue  either  a  draft  or  final  technical  report  on  the  LFT&E  of  the  Hellfire 
Missile.  IDA  did  not  prepare  a  Hellfire  Missile  technical  report  because  the 
IDA  analysts  responsible  for  the  technical  report  did  not  have  time  to  prepare 
the  report.  IDA  advised  us  that  it  would  prepare  a  Hellfire  Missile  techmcal 
report  when  and  if  time  becomes  available  even  though  DOT&E  had  already 
sent  the  live-fire  report  to  Congress. 

DOT&E,  in  the  combined  beyond  low-rate  initial  production  and  live-fire 
report,  "AH-64D  Longbow  Apache  Attack  Helicopter,"  October  1995, 
concluded  that  both  the  initial-operational  test  and  evaluation  and  the  LFT&E 
results  of  the  Longbow  Apache  and  the  Hellfire  Missile  operational  and  live-fire 
tests  were  adequate  to  provide  the  information  necessary  to  determine  their 
operational  effectiveness,  suitability,  and  survivability.  The  DOT&E  conducted 
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its  own  peer  review  of  the  Longbow  Apache  combined  beyond  low-rate  initial 
production  and  live-fire  report  before  releasing  it  to  Congress.  The  peer  review 
did  not  question  the  lack  of  IDA  technical  reports. 

C-17  Aircraft  Reporting.  The  Survivability  and  Vulnerability  Information 
Analysis  Center  prepared  die  Air  Force  LFT&E  report,  "Technical  Sunmiaiy  of 
C-17  Survivability/Vulnerability  Analysis  and  Test  Results,  Final  Report  for 
Period  June  1994  -  April  1995,"  September  1995.  DOT&E,  in  its  combined 
beyond  low-rate  initial  production  and  live-fire  report,  "C-17  Globemaster  HI 
Airlift  Aircraft,"  November  1995,  assessed  the  C-17  LFT&E  program  as 
adequate  to  support  an  assessment  of  its  vulnerability.  IDA  issued  its  final 
technical  report,  IDA  Paper  P-3093,  "Live  Fire  Test  and  Evaluation  of  the  C-17 
Aircraft,"  May  3,  1996,  about  5  months  after  the  release  of  the  DOT&E 
combined  beyond  low-rate  initial  production  and  live-fire  report  and  about 
7  months  after  the  Survivability  and  Vulnerability  Mormation  Analysis  Center 
released  its  report. 

M903  and  M430A1  Munitions  Program  Reporting.  The  Office  of  DOT&E 
stated  that  the  IDA  technical  report  will  continue  to  be  the  basis  for  reporting 
LFT&E  assessments  and  evaluations  for  programs  that  DOT&E  has  not 
designated  for  operational  test  oversight.  The  DOT&E  live-fire-test  action 
officers  are  responsible  for  preparing  the  LFT&E  reports  to  Congress.  The 
DOT&E  has  reported  on  two  such  programs,  the  M903  and  the  M430A1. 
However,  neither  of  those  munitions  programs  were  on  the  annual  test  and 
evaluation  oversight  list  even  though  they  met  the  requirements  of  Section  2366. 
For  the  M903,  die  Army  notified  DOT&E  concerning  the  development  of  the 
M903  just  before  the  production  decision.  Neither  the  Army  nor  the  Marine 
Corps  identified  the  M430A1  as  a  lethality  live-fire  candidate  until  well  after  the 
i^my  had  completed  testing  and  had  begun  production.  Consequently,  DOT&E 
did  not  exercise  LFT&E  oversight  over  the  planning  and  execution  of  those 
two  programs. 

In  March  1995,  the  Army  Materiel  Systems  Analysis  Activity  (the  Activity) 
(now  the  Evaluation  Analysis  Center)  released  its  evaluation  report,  "M430A1 
Live  Fire  Evaluation,"  to  DOT&E.  The  Activity  accomplished  its  evaluation 
report  by  reviewing  technical  test  data,  combat  data,  and  lethality  modeling  and 
analyses  that  focused  on  whether  the  M430A1  effectively  increased  legality 
against  light  armored  vehicles.  The  Activity  did  not  consider  evaluation  of  the 
fragmenting  characteristics  of  the  round  necessary.  On  May  30,  1996,  DOT&E 
released  its  LFT&E  report,  "Live  Fire  Test  and  Evaluation  Report  on  the 
M430A1  40MM  Cartridge,"  on  the  M430A1.  In  September  1996,  IDA  issued 
its  final  technical  report,  "Live  Fire  Test  and  Evaluation  of  the  M430A1  40MM 
Cartridge,"  on  the  M430A1.  The  DOT&E  LFT&E  report  and  the  IDA 
technical  report  were  substantially  identical  except  that  DOT&E  included  an 
introduction  and  summary  that  provided  an  overall  assessment  of  the  live-fire 
test  results.  The  IDA  technical  report  commented  on  test  adequacy  and  stated 
that  the  Army  did  not  conduct  the  M430A1  live-fire  test  program  in  accordance 
with  an  approved  LFT&E  strategy,  did  not  include  any  testing  against  realistic 
targets,  and  provided  only  marginally  adequate  data  for  the  lethality  assessment. 
The  DOT&E  in  its  LFT&E  report  concluded  that  the  Army  did  not  test  the 
M430A1  40  millimeter  cartridge  against  realistic  targets;  however,  the  available 
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test  data  indicated  that  the  M430A1  can  meet  its  lethality  requirements.  The 
DOT&E  released  its  LFT&E  report  about  3  months  before  IDA  released  its 
technical  report.  When  we  asked  the  IDA  project  manager  about  the  reason  for 
the  delay  in  releasing  the  report,  he  stated  that,  once  DOT&E  releases  its 
report,  its  release  takes  the  urgency  out  of  IDA  releasing  its  technical  report. 
Consequently,  DOT&E  paid  for  publication  and  distribution  of  two  M430A1 
reports  that  were  substantially  identical. 

In  May  1995,  the  Army  Materiel  Systems  Analysis  Activity  released  its 
evaluation  report,  "Live  Fire  Test  and  Evaluation  for  Caliber  .50  Saboted  Light 
Armor  Penetrator  (SLAP)  Ammunition,"  to  DOT&E.  On  September  26,  1995, 
DOT&E  issued  its  LFT&E  report,  "Live  Fire  Test  and  Evaluation  of  the  0.50 
Caliber  M903  Saboted  Light  Aimor  Penetrator  (SLAP)  Ammunition,"  to 
Congress.  The  IDA  did  not  prepare  a  technical  report  for  the  M903  live-fire 
assessment  and  evaluation  because  the  IDA  analysts  responsible  for  the  technical 
report  did  not  have  time  to  prepare  the  report. 
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Department  of  the  Army 


Command  and  Control  Vehicle.  The  Command  and  Control  Vehicle  is  an 
Acquisition  Category  II  program  that  replaces  the  M557  command  and  control 
vehicle  in  the  armored  and  mechanized  infantry  units.  The  Command  and 
Control  Vehicle  is  mounted  on  a  modified  Bradley  chassis  that  houses 
computers,  communications  equipment,  and  staff  personnel. 

M2A3  Bradley  Fighting  Vehicle.  The  M2A3  Bradley  Fighting  Vehicle  is  an 
Acquisition  Category  IC  program  that  provides  mobile  protected  transport  of 
infantry,  performs  cavalry  scout  missions,  provides  support  to  dismounted 
infantiy,  and  suppresses  or  defeats  a  wide  spectrum  of  threat  targets.  The 
Bradley  Fighting  Vehicle  system  is  an  upgrade  to  the  existing  Bradley  Fighting 
Vehicle. 

MH-47E  and  MH-60K  Special  Operations  Aircraft.  The  MH-47E  Special 
Operations  Aircraft  is  a  derivative  of  the  Boeing  CH-47  Chinook  helicopter. 
The  MH-60K  Special  Operations  Aircraft  is  a  derivative  of  the  Sikorsky  UH-60 
Black  Hawk  helicopter.  This  Acquisition  Category  II  program  provides  all- 
weather  covert  infiltration  and  exfiltration  of  Army  Special  Operations 
personnel. 

Sense  and  Destroy  Armor.  The  Sense  and  Destroy  Armor,  an  Acquisition 
Category  IC  program,  is  a  smart  submunition  designed  to  attack  self-propelled 
artilleiy.  The  Sense  and  Destroy  Armor  submunition  is  fired  from  a 
155  millimeter  howitzer,  after  which  a  parachute  deploys  to  decelerate  and 
rotate  the  submunition.  In  this  mode,  the  submunition  searches  for  targets  with 
sensors  and  then  fires  an  explosively  formed  penetrator  at  the  target. 

Wide  Area  Munition.  The  Wide  Area  Munition,  an  Acquisition  Category  n 
program,  is  an  off-route,  top-attack,  anti-vehicular  smart  munition.  The  Wide 
Area  Munition  smart  munition  uses  seismic  and  acoustic  passive  sensors  to 
search,  detect,  and  recognize  moving  targets  within  its  kill  radius.  The  Wide 
Area  Munition  smart  munition  then  launches  a  munition  toward  the  target  using 
sensors  to  detect  and  engage  the  target  with  an  explosively  formed  penetrator. 


Department  of  the  Navy 


Air-to-Air  Intercept  Missile-9X  Sidewinder.  The  Sidewinder,  an  Acquisition 
Category  ID  program,  is  the  next-generation  U.S.  short-range  air-to-air  missile. 


58 


Appendix  H.  Description  of  Major  Defense  Systems  Reviewed 


In  comparison  to  its  counterparts,  the  Sidewinder  missile  is  intended  to  have  a 
larger  operational  envelope,  better  capability  against  enemy  countermeasures, 
and  more  flexibility  for  the  aircraft  in  launching  die  missile. 

Arleigh  Burke  (DDG-51)  Class  Destroyer.  The  DDG-51  Class  Destroyer,  an 
Acquisition  Category  IC  program,  operates  offensively  and  defensively  in  multi¬ 
threat  environments.  The  DDG-51  destroyers  operate  as  units  of  carrier  battle 
and  surface  action  groups  and  as  lead  combatants  in  support  of  replenishment 
and  amphibious  groups. 

Joint  Standoff  Weapon  (Baseline  Version).  The  Joint  Standoff  Weapon,  an 
Acquisition  Category  ID  program,  is  an  air-launched,  inertially  guided,  surface 
attack,  glide  weapon  system  that  delivers  submunitions. 


Department  of  the  Air  Force 


B-2  Spirit.  The  B-2  Spirit,  an  Acquisition  Category  IC  program,  is  a  stealthy, 
long-range  bomber;  however,  its  primaiy  mission  has  evolved  from  strategic 
nuclear  to  conventional  nonnuclear  operations. 

C-130J  Hercules.  The  C-130J  Hercules,  an  Acquisition  Category  IC  program, 
primarily  performs  intratheater  airlift  missions. 

F-16  Close- Air  Support  Aircraft/Battlefield  Air  Interdiction.  The  F-16 
aircraft,  an  Acquisition  Category  ID  program,  is  a  retrofit  of  200  F-16C/D 
Block  40  aircraft  for  the  Close- Air  Support  role. 

F-22  Aircraft.  The  F-22  Aircraft,  an  Acquisition  Category  ID  pro^am,  is  to 
be  an  air  superiority  fighter  aircraft  that  will  replace  the  F- 15  Eagle  in  the  early 
2000s. 

Joint  Air-to-Surface  Standoff  Missile.  The  Joint  Air-to-Smface  Standoff 
Missile,  an  Acquisition  Category  ID  program,  is  to  satisfy  the  requirements  of 
the  terminated  Tri-Service  Standoff  Attack  Missile  Program  for  an  air-delivered 
standoff  weapon  capable  of  attacking  heavily  defended,  high-value  targets. 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Test,  Systems  Engineering  and  Evaluation 
Director,  Defense  Logistics  Studies  Information  Exchange 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Internal  Control  Officer 
Assistant  Secretary  of  Defense  (Public  Affairs) 

Director,  Operational  Test  and  Evaluation 
Deputy  Director,  Live  Fire  Test 
Director,  Administration  and  Management 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Research,  Development  and  Acquisition) 
Program  Executive  Officer,  Groimd  Combat  and  Support  Systems 
Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

Auditor  General,  Department  of  the  Amy 
Commander,  Amy  Operational  Test  and  Evaluation  Command 
Director,  Evaluation  Analysis  Center 
Commander,  Amy  Tank- Automotive  and  Amaments  Command 
Commander,  Amy  Test  and  Evaluation  Command 
Commander,  Amy  Aberdeen  Test  Center 
Director,  Amy  Research  Laboratory 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 
Program  Executive  Officer,  Surface  Combatants/ AEGIS  Program  Manager 
Program  Executive  Officer,  Tactical  Aircraft  Programs 
Director,  Acquisition  Documentation 
Auditor  General,  Department  of  Navy 

Director,  Navy  Test  and  Evaluation  and  Technology  Requirements 
Superintendent,  Naval  Postgraduate  School 
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Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Program  Executive  Officer,  Fighters  and  Bombers 
Program  Executive  Officer,  Weapons 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 
Director,  Air  Force  Test  and  Evaluation  Directorate 
Director,  Survivability  and  Vulnerability  Information  Analysis  Center 


Unified  Command 

Commander  In  Chief,  Special  Operations  Command 
Special  Operations  Command  Acquisition  Executive 
Program  Executive  Officer,  Maritime  and  Rotary  Wing 
Technology  Applications  Program  Office 

Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Lx)gistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 


Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
conunittees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 

House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 

House  Subcommittee  on  Military  Research  and  Development,  Committee  on 
National  Security 
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Part  m  -  Management  Comments 


Director,  Operational  Test  and  Evaluation, 
Comments 


OFFICE  OF  THE  SECRETARY  OF  DEFENSE 

WASHINGTON,  DC  20301-1700 

18  m  mi 

OPEHATIONAL  TEST 
AND  EVALUATION 

Mi-MORANDUM  FOR  THE  INSPECTOR  GENERAL,  DEPARTMENT  OF  DEFENSE 

SUBJECT:  Draft  of  a  Proposed  Audit  Report  on  Live-Fire  Test  and  Evaluation  of  Major 
Defense  Systems 

Thank  you  for  the  opportunity  to  review  your  draft  audit  report  of  the  T-ive  Fire  Test  and 
hValuation  of  Msgor  Defense  Programs.  I  have  completed  my  review  and  appreciate  the  efforts  of 
the  DoD  IG  team  to  address  the  program*  its  operation,  and  impact. 

Before  discussing  the  four  specific  findings  presented  in  the  draft,  1  would  note  that  the 
overall  conclusion  of  the  team  was  that  “/fte  Office  of  the  Secretary  of  Defense  and  the  Military 
Departments  effectively  Implemented  the  LFTdcE  process  from  the  programs  reviewed^*  and  we 
certainly  agree.  1  am  also  pleased  that  the  team  pointed  out  that  significant  efficiencies  have  been 
brought  about  by  the  FASA-mandated  move  of  the  LFT&E  program  to  this  office.  'I’hc  move  has, 
indeed,  brought  about  not  only  economics  of  scale  and  more  efficient  management  and  reporting  to 
the  Secretary  of  Defense  and  to  the  Congress,  but  it  has  also  brought  about  a  much  better 
integration  of  vulnerability/lethality  issues  with  issues  of  concern  to  the  OT&E  community.  Ihe 
draft  report  ^so  note.s  that  the  Live  Fire  Test  and  Evaluation  Office  has  **empUjyed a  number  of 
creative  innovatiom  to  improve  awareness  of  the  LKIkScE  process something  we  will  continue  to 
pursue. 

To  keep  my  response  brief,  I  will  confine  my  comments  to  the  four  specific  findings 
themselves  and  will  not  address  tlie  detailed  comments  made  in  the  many  pages  of  discussion. 

Finding  A>  .Live  Fire  Test  and  Evaluation  Oversight  List.  Partially  concur. 
Maintenance  of  the  OSD  T&E  Oversight  List  is  a  constant  and  recurring  task  and  by  DoD  5000 
Regulation,  must  be  updated  at  least  annually.  Determining  candidacy  of  systems  for  T&E 
oversight  inclusion  is  more  difficult  than  it  may  first  appear.  Not  only  does  one  have  to  identify 
the  potential  candidates  as  they  begin  their  acquisition  journey,  sufficient  details  need  to  be  known 
about  its  anticipated  usage  to  understand  whether  It  will  be  expected  to  provide  protection  to  the 
user,  to  know  whether  its  total  cost  will  exceed  the  dollar  thresholds  established  for  candidacy,  or, 
if  an  upgrade  which  docs  not  qualify  as  a  new  system  due  to  its  funding  level,  whether  the  upgrade 
would  significantly  affect  the  survivability  or  lethalify  of  the  system  being  upgraded. 

The  DoD  5000.2R  gives  specific  responsibility  to  the  Services  to  step  forward  and  begin 
the  process  and  identif>'  potential  systems  which  may  qualify  for  L1*T&E  oversight.  T  agree  that 
this  responsibility  has  not  been  exercised  uniformly  within  the  Services.  The  same  document  also 
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explicitly  gives  the  DOT&E  the  responsibility  to  approve  LFT&E  candidates  for  oversight.  In 
practice,  what  has  happened,  as  your  draft  report  has  concluded,  is  that  the  identiilcation  of 
T.FT&E  candidates,  especially  for  ACAT 11  and  below,  has  been  less  than  efficient  and  we  have 
had  to  rely  on  any  number  of  sources  to  obtain  information  to  identify  potential  candidates  when 
the  Services  have  not  stepped  forward  with  nominations  themselves.  I  concur  with  your 
recommendation  that  the  Services  comply  with  current  DoO  regulations  requiting  the  Services  to 
nominate  potential  candidates  for  LFT&E  consideration.  I  concur  with  your  recommendation  that 
the  draft  oversight  list  should  be  circulated  to  the  Services  for  their  review  and  comment  prior  to 
publicatiom  I  also  concur  with  your  recommendation  that  ACAT  IT  and  TTT  program  lists  be 
provided  to  this  office  on  a  regular  basis  for  review. 

Finding  B.  Reporting  Live-Fire  Test  and  Evaluation  Results.  Partially  concur.  In 
response  to  the  DoD  TO  draft  report  recommendation,  I  have  removed  the  requirement  from  our 
renewal  contract  with  the  Institute  of  Defense  Analyses  (IDA)  to  have  such  an  LFT&E  technical 
report  as  a  mandatory  deliverable  for  cvcr>'  program. 

One  of  the  major  deliverables  required  by  statute  is  for  the  Director,  Operational  Test  and 
Evaluation,  to  prepare  the  independent  Live  Fire  Test  and  Evaluation  Report  for  the  Secretary  and 
the  four  defense-related  committees  of  the  Congress.  Hcncc,  wc  place  a  heavy'  emphasis  on  how'  it 
is  prepared,  ^^o  helps  in  its  preparation,  its  clarity  and  completeness  and  most  of  all,  its  accuracy 
and  timeliness.  IDA,  since  the  inception  of  the  Live  Fire  Test  program  over  a  decade  ago,  has  been 
the  exclusive  support  contractor  in  assisting  the  LFT&E  cilice  in  its  preparation  of  these  reports. 
Since  the  beginning,  these  reports  have  been  candid,  comprehensive,  and  technically  sound.  With 
the  move  of  the  LFT&E  mission  to  the  DOT&E  office,  wc  have  fell,  as  the  Congress  in  its  FASA 
legislation  and  your  draft  report  both  conclude,  that  considering  LFT&E  results  in  the  context  of 
Ol'&E  results  and  OT&E  results  in  the  context  of  LFT&E  makes  a  lot  of  sense.  Hence,  for  those 
programs  which  require  both  a  Beyond  Low-Rate  Initial  Production  OT&E  Report  as  well  as  a 
Live  Fire  Test  and  Evaluation  Report,  we  prepare  and  submit  these  two  reports  often  under  one 
cover,  thereby  enabling  a  more  ellicient,  balanced,  and  complete  assessment  of  the  system  under 
review.  Flowever,  to  enable  the  combined  r^rts  to  be  more  concise  and  readable,  a  significant 
amount  of  tedmical  material  historically  included  in  LFT&E  Reports  when  they  had  been 
submitted  alone,  must  be  removed.  This  removed  material,  nevertheless,  is  critical  to  the  overall 
conclusions  drawn  in  the  combined  reports  submitted  to  the  Congress  and  should  be  retained  for 
future  reference. 


The  comment  was  made  in  the  draft  report  that  these  technical  reports  are  often  published 
long  after  the  LFT&E  Report  goes  to  the  Congress.  This  is  coinect  There  is  no  need  for  the 
technical  report  to  be  published  prior  to  the  congre.ssiona1  submission,  llic  only  requirement  is 
that  the  technical  work  be  performed  to  draw  the  correct  conclusions  prior  to  submission.  The 
technical  rqwrts,  which  can  be  published  on  a  less  demanding  schedule,  contain  vital  information 
which  diould  be  retained  for  future  use  and  reference,  especially  recognizing  that  an  increasing 
share  of  systems  on  the  oversight  list  arc  upgrades  to  current  systems.  'Jlus  iitformaiion  will  serve 
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as  a  vital  baseline  for  comparison  mih  future  upgrade  systems  on  oversight.  As  a  result,  we  plan 
to  maintain  these  files  for  future  reference. 

Finding  C>  Processing  Requests  for  Waiver  From  Full-Up,  System-Level 
Survivability  or  Lethality  Tests,  Concur.  This  finding  is  certainly  appropriate  and  is  an  jssuc 
which  we  have  wrestled  with  since  the  DoD  5000  was  reissued  nearly  two  years  ago.  The  current 
confusion  in  the  DoD  5000  series  over  how  to  obtain  a  waiver  stems,  in  pari,  from  a  policy  letter 
which  was  issued  by  the  DepSecDcf  in  1995  prior  to  the  leissuance  of  the  DoD  5000  series.  This 
letter  was  then  translated  into  the  DoD  5000  scries  and  confused  the  process  even  more.  I  agree 
that  the  waiver  process  ^ould  be  clarified  and  made  consistent 

In  losponse  to  this,  we  have  recently  published  portions  of  our  DOT&E  Notebook  w'Hch 
include  these  procedures.  I'he  waiver  process  is  fully  explained  and  is  consistent  Following  this, 
we  plan  to  submit  to  the  DoD  5000  Defense  Acquisition  Policy  Working  Group  a  revised 
regulation  for  consideration  and  inclusion. 

Finding  D.  Strengthening  Management  Controls,  Partially  concur.  As  your  study 
concluded,  the  ^^LFTdcE  office  has  employed  a  number  of  creative  innovations  (o  bnprove 
awareness  of  the  LFT&E  process . . .  Aar  written  a  number  of  papers,  conducted  LFT&E 
educational  conferences,  and  established  an  Internet  website . , . Over  and  above  this,  we  have 
also  produced  two  cducatiomd  videos,  hundreds  of  copies  of  which  are  being  used  nationally  to 
communicate  wliat  is  expected  in  the  LFT&E  program.  Furthermore,  we  wrote  and  published  an 
LFT&E  Users’  Guide  over  eight  yeans  ago  and  circulated  it  widely  as  a  training  aid.  Ibis  Users’ 
Guide  is  being  supplanted  by  the  LFf&E  website  and  the  DOT&E  Notebook  togefiier  providing 
much  more  detail  on  how  lo  plan  and  execute  LF1*&E  than  ever  before. 

It  IS  also  noteworthy  that  there  is  more  descriptive  material  on  Live  Fire  Testing  in  the 
DOD  5000  series  of  Directives  and  Regulations  than  for  nearly  any  other  subject,  and  certainly 
more  than  cither  developmental  testing  or  operational  testing.  All  of  these  aforementioned 
resources  arc  also  used  to  help  train  the  LFT  staff,  including  the  one-year  rotational  trainees  serving 
in  the  office  who  make  up  a  significant  portion  of  the  LFf&E  workforce.  Hence,  I  can  only 
partially  concur  with  the  finding  that  LFT&E  administrative  policies  and  procedures  are  not  well- 
known  and  established. 

In  response  Id  your  recommendations,  we  will  review  LFT&E  administrative  procedures 
relating  to  the  documentation  and  support  of  conclusions  and  recommendations  made  for  LFT&E 
program  oversight.  Also,  in  response  to  the  recommendations  of  your  draft  report,  we  have 
already  accelerated  the  completion  of  our  DOT&E  Notebook.  In  lact,  the  full  integration  of 
LFT&E  policy  and  procedures  has  already  been  completed  in  draft  and  should  be  published  in 
advance  of  September  30,  the  date  recommend  in  your  draft  report. 
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1  must  note  that  we  are  taking  all  these  actions  (in  addition  to  the  regular  day-to-day 
demands  on  the  LFT&E  Office)  in  an  office  which  has  only  five  full-time  professionals. 

In  conclusion,  I  again  want  to  commend  the  DoD  IG  team  for  a  professional  and  thorough 
job  on  this  study,  The  entire  defense  community  will  benefit  from  their  efforts. 

Philip  E.  Coyle 
Director 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 

2  2  /lUS  7997 


MEMOR  ANr>UM  FOR  Ol-  I'lCli  OF  THE  INSPECTOR  GENERAL 

I- ROM:  DIRECTOR,  TEST  SYSTEMS  ENGINEERING,  AND  KVAI..UAT10N 

SUBJECT:  Review  of  the  Draft  DO  DIG  Audit  Report,  “Live-Fire  Test  and  Evaluation  of  Major 
Defense  Systems"',  (Project  No.  6AE-0067.00),  June  18, 1997 


'lliis  office  has  reviewed  the  subject  draft  report  and  provides  the  following  comments; 

•  Eindina  A.  Concur  with  the  recommended  improvements  in  developing  the  OSD  Test  and 
Evaluation  Oversight  last 

•  Finding  13.  Partially  concur  tvith  this  linding  which  recommends  Ihe  lerminalion  of  formal 
Ll’T&E  technical  reports  prepared  by  the  Institute  for  Defense  Analyses  (IDA)  since  the  results 
arc  included  in  DOT&E’s  Beyond  LRIP  Reports  to  Congress.  How'cver,  it  is  our  view  that  the 
DOT&E  should  have  the  flexibility  to  direct  the  preparation  of  separate  technical  reports,  wlicn 
they  appear  to  have  some  future  value.  This  is  especially  true  given  the  trend  to  use  past  test 
data  to  the  greatest  degree  possible  to  reduce  the  costs  of  future  tests.  These  reports  include  the 
unbiased,  technical  analyses  which  are  used  by  the  DO  lVfeli  action  officers  in  producing  the 
Live  Fire  “Evaluation”,  which  may  include  more  than  just  the  technical  analysis. 

•  Finding  C.  Concur  with  the  recommendation  to  revise  DODR  5000.2-R  to  describe  a  consistent 
LFT&E  waiver  process.  On  page  20,  the  repon  states  that  the  OASN(RDA)  believed  that 

wm  vers  could  not  be  submitted  alter  a  program  piissed  MS  11,  implying  that  this  was  a 
misunderstanding  of  sec.  2366  by  OASN.  However,  the  report  does  not  clarify  that  implied 
misperception.  Was  this  not,  in  fact,  a  correct  interpretation  by  OASN? 

•  Finding  D.  No  comments  are  provided  on  the  recoiuniend aliens  concerning  DOT&E’s  I.TT&R 
inlemal  ••idministnitivc  policies. 

DTSE&K's  Point  of  contact  for  these  actions  is  Mr.  Richard  I.edesma,  (703)  695-4421, 


Patricia  Sanders 
Director,  Test,  Systems 
Engineering  and  Evaluation 
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DERARTME^^‘  OF  THE  ARMY 
DEPUTY  UMDCn  ■ECRCTAflV  OP  THE  AAUV 
OPKHAHONp  PIEWAROH 
1  m  Aay  Y  FEMnooM 

WAomNonpor*  dc  2091DOIM 


**  WG  m 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  DEPARTMENT  OF 

DEFENSE  (ATTN;  ACQUISITION  MANAGEMENT) 

SUBJECT :  Audit  Report  on  Live-Fire  Test  and  Evaluation  of  Maj or  Defense 
Systems  (Project  No.  6AE-0067,00) 


The  Department  of  the  Army  concurs  with  the  findings  and 
recommendations  of  the  subject  report.  Recommendation  A.1  calls  for  the  Deputy- 
Under  Secretary  of  the  Army  (Operations  Research)  and  appropriate  offices  of 
other  services  to  'formally  ^cument  and  recommend  candidate  programs  to  the 
Director*  Operational  Test  and  Evaluation,  for  LFT&E  oversight.”  The  Army  will 
join  in  whatever  procedure  the  DOT&E  establishes  for  this  action. 

Walter  W.  Hollis 

Deputy  Under  Secretary  of  the  Army 
(Operations  Research) 
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DEPARTMENT  OF  THE  AIR  FORCE 
WMSHINqTON,  DC 


II  m 


MEMORANDUM  FOR  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 


FROM:  SAF/AQ 

1060  Air  Force  Pentagon 
Wadbington  DC  20330^1060 

SUBJECT:  “Live-Fire  Test  and  Evaluation  of  Ni^'or  Deiense  Systems,”  Jnnc  18, 1997,  Project 
No.  6AE-0067,00 

1 .  This  is  in  reply  to  your  memoranidum  requesting  the  Assistant  Secretary  of  the  Air  Force 
(Financial  Management  and  ComptcoUer)  to  provide  Air  Force  comments  on  subject  rq>ort,  We 
have  the  following  commencs  and  recommendations; 

Recommendation  A  .1 .  ”...  foimalty  document  and  recommend  candidate  programs  to  the 
Director,  Operational  Test  and  Oversight,  for  LFT&B  oversight.”  Response:  Concur.  Although 
API  99-105  provides  guidance  and  procedures  forihe  LFT&E  of  Air  Force  systems,  the 
document  does  not  provide  the  specific  guidance  on  the  *  Vben,  how,  and  who**  of  identifying 
caiuiidaJe  programs.  AF/IEP  and  SAF/AQX  will  update  this  document  to  provide  the  specific 
guidance  required  to  systematically  document  and  recommend  our  candidate  programs.  The 
estimated  completion  date  for  this  update  is  Apr  98. 

Recommendation  C.2.  . .  F-22  program  is  required  to  submit  a  request  for  waiver  from 

friU-up,  system-level  survivability  testing  by  30  Sep  97.”  Response:  Concur.  We  intend  to  meet 
the  30  Sep  97  submittal  date. 

Comment:  On  page  50,  your  description  of  the  B-2  bomber  diould  be  corrected  for  acctnacy. 
The  description  should  read;  The  B-2  Spirit,  an  Acqulaition  Category  IC  program,  is  a  atealtfay, 
long-range  bomber;  however,  its  primary  mission  has  evolved  from  strategic  nuclear  to 
conventional  nonnuclear  operations, 

2.  In  addition  to  my  comments  listed  i^ve,  AF/TE  has  additional  comments  (Atch  1)  that  Fm 
forwarding  to  you  in  their  entirety. 

3.  The  point  of  contact  for  fiirther  questions  on  this  review  from  our  office  is  Maj  Daniels, 
697-7505,  e-mail  address  daniel8j@af.pentagon.inil. 
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DEPARTMENT  OF  THE  AIR  FORCE 

HCADQUARTCRS  UNITED  STATE*  AIR  FORCE 


14  MIS  m 

MEMORANDUM  FOR  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENE3RAL 
DEPARTMENT  OF  DEFENSE 

FROM:  HQUSAF/IE 

1650  Air  Force  Pentagon 
Washington,  DC  20330-1650 

SUBJECT:  DoDIO  Draft  Report,  live  Fire  Teat  and  Evaluation  of  Major  Deftnae 
Systame,  (Pttaect  6AE-00e7.00) 

This  it  in  reply  to  your  mamorandum  retiueeting  the  Aaaiitant  Secretary  of  tho 
Air  Poica  (Financial  Management  and  Gonqptroller)  to  provide  Air  Force  commente 
on  subject  report.  We  have  the  following  eommenta  and  recommendetione: 

Finding  A;  Live  Fire  Teat  and  Evaluation  Oversight  List 

a.  We  non-concur  with  recommendation  A2.a  which  suggeete  using 
ACAT  in  program  liete  to  help  identify  candidate  covered  ayetems  subject  to  LFT*B 
and  DOT&E  oversifijrt.  We  do  not  believe  a  review  of  ACAT  III  program  liate  will  aid 
the  seareh  for  candidate  LFT&E  eyetems  covered  by  Title  10  section  2366.  However, 
we  concur  with  the  use  of  ACAT  11  lists.  RATIONALE:  Title  10  describee  which 
programs  are  eubject  to  LFT&E,  and  speciEcally  targets  ACAT  1  and  ACAT  II 
programs.  According  to  the  definition  “covered  system”  in  §2366(e)(l),  and  tiie 
definition  of  “major  system”  in  eeetlon  2802(6),  only  ACAT  I  and  H  programs  are 
covered.  ACAT  in  programs  are  not  potential  LFT&E  candidates,  nor  are  they 
subject  to  OSD  LFT&E  oversight. 

b.  We  concur  with  Recommendations  A,t,  A2.b,  and  A.3. 

Finding  B:  Reporting  live  Fire  Test  and  Evaluation  Results 

a.  On  page  13,  the  draft  makes  refersnee  to  a  “DOT&E  ind^endent  repent 
to  Congress.”  According  to  Title  10  eection  2366(d),  the  Secretary  of  Defense,  not  the 
DOT&E,  submits  the  LFT&E  report  to  Congress.  Independent  reporting  firom 
DOT&E  is  only  reguired  for  initial  eperationai  test  and  evaluatfon  (lOT&E) 
described  in  section  2399,  but  is  not  required  for  LFT&E  reports  in  section  2366.  By 
the  LFT&E  and  OT&E  reports,  and  sending  both  directly  to  Congress 
without  intervening  SECDEF  review,  the  SECDEF s  control  over  the  LFT&E  reporl^s 
contents  before  eubmiasion  is  compromised. 
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h.  We  non-^^oncur  with  reconunendation  to  finding  B.  The  BLRIF  report 
and  the  LFT&B  report  must  remain  separate  to  ensure  the  SBCDEF  can  exercise  his 
authority  to  edit  the  LFT&E  report  before  it  is  submitted  bv  the  SECDEF  to 
Congress  (as  mandated  in  Section  2366). 

Finding  C,  Processing  Requests  for  Waiver  from  Full-Upt  System'^Level 
Survivability  or  Lethality  Tests 

a.  On  page  20»  first  full  paragraph,  the  report  preaumea  the  DDG-51  is 
subject  to  LFT&E  despite  the  determination  by  the  Office  of  the  Assistant  Secretary 
of  the  Navy  (RD&A)  that  the  system  was  no£  a  covered  83ntem.  The  paragraph 
should  be  rephrased  to  make  clear  that  this  is  the  opinion  of  the  DoOIG  only. 
RATIONALE:  The  statement  is  incorrect.  The  issue  of  which  systems  are  subject  to 
LFT&E  (e.g.,  are  ‘'covered  systems*')  is  currently  being  debated  among  senior  DoD 
leaders.  A  number  of  other  systems  besides  the  DDG-61,  euch  as  the  C-130  J  aircraft, 
and  the  CH-47D  and  UH-GOL  helicopters,  are  potentially  impacted  by  the  outcome  of 
those  diacuesione.  DoD  5000.2*R,  Appendix  IV,  concedes  the  difficulty  of  resolving 
this  issue  when  it  states,  "Legal  counsel  should  be  consulted  to  veri^  the  final 
determination  of  program  statue  ”  In  short,  simple  generalisations  about  the  covered 
statue  of  systems  is  misleading.  In  the  C*130J'e  case,  the  Air  Force  asked  the  OSD 
General  Counsel  (OSD/GC)  their  opinion  about  the  covered  status  of  the  C-ISOJ, 
and  OSD/GC  said  the  law  may  not  apply  to  the  C-130  J  program, 

b.  We  non-concur  with  recommendation  Cl.  Unless  the  DDG-51 
destroyer  is  identified  as  a  covered  system,  a  waiver  to  the  statute  is  not  required. 
We  concur  with  recommendations  02  and  C3. 

Finding  D.  Strengthening  Management  Controls,  We  concur  with  the 
recommendations. 

Additional  minor  comments  on  the  draft  report  are  attached.  My  points  of 
contact  are  Maj  Bo  Tye  and  Chuck  Iriska,  (703)  695-0900,  and  fax  (703) 
695-0603. 

Lt  Gen,  USAF  (RetiM®r"^ 

Director,  Teat  and^^uation 

Attachment: 

Additional  Comments 
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SPECIFIC  COMMENTS 
on  th« 

DoDIG^s  REPORT  ON  LIVE  FIRE  TEST  &  EVALUATION  (LFT&E) 


-  Beginning  on  page  2,  second  paragraph,  and  thronghout  the  report:  Th«  report 
uaea  the  vrxmg  termincdogy  when  describing  the  ^rpee  of  syateme  covered  h^ 
Title  10  aection  2366.  The  term  '^tniyor  syatem**  should  be  used  instead  of”  major 
Defense  syetem.’'  EATIOKM/E:  Section  2366  addresaes  only  ^mvtioT  aystems.” 

-  On  page  3,  the  draft  report  applauds  DOT&E  for  its  innovative  use  of  *an 
internet  webeite  to  inform  Individuals  involved  in  LFT&E  about  the  latest 
LFT&E  advances,  guidance,  and  issues."  In  the  eame  vein,  the  DoDIG's  report 
should  recommend  that  DOT&E  put  thm  LFTAE  Oversight  List  on  that  website 
for  review  and  comment  before  the  list  is  ofiElcially  published. 

-  Page  9,  second  paragraph:  Delete  the  statement  about  the  SECDEF  being  *in 
the  process  of  revising  DoD  Regulation  6000,2-R  to  require  commercial  items, 
including  commercial  ofif-the-sbelf  items,  nondevelopmental  items,  and  advanced 
concept  technology  demonstrators  or  prototypes  that  meet  the  definition  of  a 
covered  system  or  mejor  jzmnitiona  program  to  be  subject  to  LFTAE.* 
RATIONALE:  This  change  has  been  withdrawn, 

-  Page  14,  in  the  C-I7  subparagraph:  Recommend  adding  the  following  sentence 
to  the  report:  "Despite  several  requests,  IDA’s  technical  report  on  C-17  LFT&E 
has  not  been  released  to  the  Air  Force  for  review,  nearly  two  years  after  the  Oct 
95  Defense  Acquisttion  Board  decision  to  proceed  beyond  LRIP,  This  report  may 
contain  valuable  lesaozu  learned  which  could  help  improve  other  Air  Force 
programs." 

.  Page  33,  last  paragraph:  Ihe  text  mistakenly  states  that  the  Federal 
Acquisition  Streamlining  Act  (FAS A)  "placed  LFT&E  under  DOT&E,”  FASA 
only  added  a  provision  to  Title  10  section  139  requiring  the  Director  to  “monitor 
and  review  the  live  fire  testing  activities  of  the  Department  of  Defense  provided 
for  under  section  2366."  It  was  the  DoD  who  moved  the  LFT&E  office  and 
personnel  from  OSD/DTSE&E  to  OSD^OT&£. 

^  •  Page  44,  paragraph  2;  The  text  contains  some  arrora  regarding  the  extent  of 
DOT&E’s  responBibilities  for  LFT&E  as  directed  in  FASA  and  Title  10,  The  draft 
states  "DOT&E  accomplishes  its  oversight  responsibilities  by  reviewing  and 
approviji^  live  fire  documents  . , More  correctly.  Title  10  section  139(b)(6)  only 
gives  the  Director  authority  to  ^monitor  and  review  the  live  fire  testing 
activities"  of  the  DoD.  (it  Jics  added)  In  line  7,  DOT&E’s  authority  extends  only 
to  approving  the  LFT&E  strategy  and  not  the  LfT&£  plans.  Finally,  change  the 
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last  sentence  to  read,  DOTdsE  oversees  the  live  fire  test  and  evaluation  of  a 
program  . . .  DOT&E  does  not  conduct  LFT&E  because  they  do  not  have  the 
specialised  assets  or  personnel  for  this  kind  of  woork. 

-  Include  a  verbatim  copy  of  $2366  as  an  appendix  to  the  report.  RATIONALE: 
The  reader  should  have  actual  law  available  for  reference  due  to  the 
complexity  of  ‘^e  issues^ 
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Audit  Team  Members 


The  Acquisition  Management  Directorate,  Office  of  the  Assistant  Inspector 
General  for  Auditing,  DoD,  prepared  this  report. 
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INTERNET  DOCUMENT  INFORMATION  FORM 


A  .  Report  Title:  Live-Fire  Test  and  Evaluation  of  Major  Defense  Systems 


B.  DATE  Report  Downloaded  From  the  Internet:  10/11/99 


C.  Report’s  Point  of  Contact:  (Name,  Organization,  Address,  Office 
Symbol,  &  Ph  #):  OAIG-AUD  (ATTN:  AFTS  Audit  Suggestions) 

Inspector  General,  Department  of  Defense 
400  Army  Navy  Drive  (Room  801) 

Arlington,  VA  22202-2884 


D.  Currently  Applicable  Classification  Level:  Unclassified 

E.  Distribution  Statement  A:  Approved  for  Public  Release 

F.  The  foregoing  information  was  compiled  and  provided  by: 

DTIC-OCA,  Initials: _ ^VM _ Preparation  Date  10/11/99 


The  foregoing  information  should  exactly  correspond  to  the  Title,  Report  Number,  and  the  Date 
the  accompanying  report  document.  If  there  are  mismatches,  or  other  questions,  contact  the 
above  OCA  Representative  for  resolution. 
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